3d manipulation using applied pressure

ABSTRACT

Placement by one or more input mechanisms of a touch point on a multi-touch display device that is displaying a three-dimensional object is detected. A two-dimensional location of the touch point on the multi-touch display device is determined, and the touch point is matched with a three-dimensional contact point on a surface of the three-dimensional object that is projected for display onto the image plane of the camera at the two-dimensional location of the touch point. A change in applied pressure at the touch point is detected, and a target depth value for the contact point is determined based on the change in applied pressure. A solver is used to calculate a three-dimensional transformation of the three-dimensional object using an algorithm that reduces a difference between a depth value of the contact point after object transformation and the target depth value.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/165,853, filed on Apr. 1, 2009 and entitled “Screen-SpaceFormulation for 2D and 3D Manipulation,” and U.S. ProvisionalApplication Ser. No. 61/248,670, filed on Oct. 5, 2009 and entitled“Screen-Space Formulation for 2D and 3D Manipulation,” both of which areincorporated herein by reference in their entirety for all purposes.

BACKGROUND

Some touch screen device types, such as touch enabled computer monitors,personal digital assistants, media players, and cellular telephones,enable users to interact with onscreen elements by touching a displaywith a finger or a stylus. Single-touch devices are capable ofrecognizing a single point of contact, and they enable users to performa variety of two-dimensional (2D) interactions such as moving on-screenobjects through a so-called “dragging” operation.

More recently, devices have emerged that recognize more than onesimultaneous point of contact for interacting with 2D on-screen objects.Such so-called “multi-touch” interfaces generally support single-touchinteractions, as well as multiple-touch (generally two-fingered)gestures. These multi-touch gestures are sometimes known asrotate-scale-translate (RST) interactions because these interactionsgenerally allow a user to spin, slide, or resize 2D objects in a 2Dplane. One commonly used multi-touch gesture involves a user touching a2D object at two or more points in a display and then revolving thosepoints relative to each other to effect virtual rotation of the object,similar to the manner in which a person could spin a photograph or apiece of paper lying on a tabletop. Another commonly used multi-touchgesture involves a user touching a 2D object at two points on a display,and then spreading or pinching the gap between the two points to enlargeor shrink the visible size of the object.

SUMMARY

In one general aspect, placement by one or more input mechanisms of atouch point on a multi-touch display device that is displaying athree-dimensional object is detected. A two-dimensional location of thetouch point on the multi-touch display device is determined, and thetouch point is matched with a three-dimensional contact point on asurface of the three-dimensional object that is projected for displayonto the image plane of the camera at the two-dimensional location ofthe touch point. A change in applied pressure at the touch point isdetected, and a target depth value for the contact point is determinedbased on the change in applied pressure. A solver is used to calculate athree-dimensional transformation of the three-dimensional object usingan algorithm that reduces a difference between a depth value of thecontact point after object transformation and the target depth value.

The various aspects, implementations, and features disclosed may beimplemented using, for example, one or more of a method, an apparatus, asystem, tool, or processing device for performing a method, a program orother set of instructions, an apparatus that includes a program or a setof instructions, and a computer program stored on a tangible,computer-readable storage medium. The tangible, computer-readablestorage medium may include, for example, instructions that, whenexecuted, cause a computer to perform acts specified by theinstructions.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and the drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a sequence of diagrams that illustrate a user manipulation ofa 3D object displayed on a 2D display interface.

FIGS. 2A-2F are a sequence of diagrams that illustrate a usermanipulation of a 3D object displayed on a 2D display interface, asviewed from a perspective of the user and from an oblique world-spaceperspective.

FIG. 3 is a flow diagram of a process for enabling a user to manipulate3D objects through a 2D display interface.

FIGS. 4A-4D are each a sequence of diagrams that illustrate gestures foruser manipulation of a 3D object displayed on a 2D display interface.

FIGS. 5A-5E are each a sequence of diagrams that illustrate a usermanipulation of a 3D object displayed on a 2D display interface thatimparts an error, as viewed from a perspective of the user and from anoblique world-space perspective.

FIGS. 6A-6B are diagrams that illustrate a user manipulation of a 3Dobject displayed on a 2D display interface that may induce ambiguousrotation, as viewed from a perspective of the user and from an obliqueworld-space perspective.

FIG. 7 is a diagram that illustrates a user manipulation of a 3D objectdisplayed on a 2D display interface that leads to ambiguous rotations ofthe 3D object.

FIGS. 8( i)-8(v) are a sequence of diagrams that illustrate a usermanipulation of a 3D object displayed on a 2D display interface thatresults in a rotational exhaustion situation, as viewed from aperspective of the user.

FIGS. 9( i)-9(v) are a sequence of diagrams that illustrate a usermanipulation of a 3D object displayed on a 2D display interface thatdoes not result in a rotational exhaustion situation, as viewed from aperspective of the user.

FIGS. 10A and 10B are a flow diagram of a process for enabling a user tomanipulate 3D objects through a 2D display interface that includesdetection and correction of rotational exhaustion situations.

FIG. 11 is a flow diagram of a process for detecting a rotationalexhaustion situation.

FIGS. 12A-12C are a sequence of diagrams that illustrate differentdistributions of error among touch points placed on a 2D displayinterface.

FIG. 13 is a flow diagram of a process for enabling a user to manipulate3D objects through a 2D display interface that distributes error amongtouch points in a controlled manner.

FIG. 14 is a sequence of diagrams that illustrate a user manipulation ofa 3D object displayed on a 2D display interface where the motion of the3D object is restricted by a penalty, as viewed from a perspective ofthe user.

FIG. 15 is a sequence of diagrams that illustrate a user manipulation ofa 3D object displayed on a 2D display interface where the motion of the3D object is restricted by multiple penalties, as viewed from aperspective of the user.

FIG. 16 is a flow diagram of a process for enabling a user to manipulate3D objects through a 2D display interface that restricts motion of the3D objects using penalties.

FIG. 17 is a sequence of diagrams that illustrate a user navigation intoa complex 3D object displayed on a 2D display interface.

FIG. 18 is a sequence of diagrams that illustrate a one touch pointinteraction to navigate over the surface of a simulated 3D globe.

FIG. 19 is a sequence of diagrams that illustrate a two touch pointinteraction to navigate over the surface of a simulated 3D globe.

DETAILED DESCRIPTION

A 2D display interface device (e.g., a multi-touch capable displayinterface device) enables users to manipulate a simulatedthree-dimensional (3D) object displayed on a 2D display interface by“touching” one or more points on the displayed representation of the 3Dobject, and then moving those touch points about in the generally 2Dplane of the interface. The user may “touch” the one or more points onthe displayed representation of the 3D object by physically contactingthe corresponding points in the 2D display interface with a finger, astylus or some other input mechanism and/or by hovering the finger, thestylus or the other input mechanism around, or otherwise in the vicinityof, the corresponding points in the 2D display interface. The 2D displayinterface device determines the 2D positions of the touch points andidentifies the corresponding 3D points “touched” on the 3D object (i.e.,the contact points on the surface of the 3D object that are displayeddirectly underneath or substantially underneath the touch points placedon the 2D display interface). As the user moves the touch points aboutthe substantially 2D interface, the 2D display interface devicesrotates, scales, and/or translates the 3D object in order tosubstantially maintain a display of the same 3D contact points under thesame, respective, 2D touch points. Typically, the moving by the user ofa touch point from one location to another on the 2D display interfaceis accomplished by the user moving the touch point while maintaining asubstantially continuous “touch” of the screen.

One appeal of a 2D display interface device that enables usermanipulation of objects in the above-described manner is that the usermay be given the impression that the user is directly manipulating the3D object in the same way that he/she would manipulate an object in thereal world (i.e., the nom-simulated world). For example, if the useruses his/her fingers to “touch” on object displayed on the interface,die user can slide his/her fingers along the surface of the 2D displayinterface, and the 2D display interface device reacts by manipulating(e.g., rotating, translating, and scaling) the 3D object so that thesame point on the surface of the object remains substantially underneaththe same fingertip. Since the user is thereby able to manipulate theobject so that it moves in a predictable and realistic fashion, the usermay be given the impression of “gripping” and manipulating a real object(i.e., a non-simulated 3D object). Direct manipulation can essentiallyprovide an intuitive and controllable mapping between points in anobject's local space (i.e., points in object-space) and points on thescreen of the 2D display interface (i.e., points in screen-space),without the need for any explicit gesture processing.

Referring now to FIG. 1, a sequence of images 100 a-100 f illustrate anexample of a direct manipulation of a simulated 3D object, a cube 102,as it may be displayed on a 2D multi-touch enabled display interface ofa computer. In the image 100 a, the cube 102 is displayed in an initialposition, wherein the user touches the interface with his or her fingersat a touch point 104, a touch point 106, and a touch point 108. Each ofthe touch points 104-108 has a corresponding contact point, which is thelocation on the surface of the 3D object that is displayed at the giventouch point. Examples of touch points and corresponding contact pointswill be described further below in reference to FIGS. 2A-2F.

As is shown sequentially in the images 100 b-100 d, the user moves thetouch points 104, 106 and 108 across the 2D display interface by slidinghis/her fingers across the 2D plane of the interface. The cube 102 isanimated by the 2D display interface device to rotate, translate and/orscale the cube 102 such that the touch points appear to remainsubstantially at the same contact points on the cube 102 as the user'sfingers move across the interface. In the image 100 f, the user'sfingers have arrived at a new location 100 for touch point 104, a newlocation 112 for touch point 106, and a new location 114 for touch point108, and the cube 102 is brought into a corresponding orientation (i.e.,a location and rotational orientation) seed that the same contact pointsremain displayed substantially underneath their respective touch pointsat the new locations 110-114.

FIGS. 2A-2F are a sequence of diagrams that illustrate a directmanipulation of a 3D object by a user through the placement and movementof touch points on a 2D display interface. FIG. 2A shows the user havingplaced three touch points 206 a, 208 a and 210 a on a 2D displayinterface 202 that appear to touch a simulated cube 204. The view shownin FIG. 2A is taken from the user's perspective. The location of each ofthe touch points 206 a, 208 a and 210 a can be represented inscreen-space.

Screen-space is a 2D space that is used to represent locations on the 2Ddisplay interface (i.e., on the “screen”). A location in screen-spacemay be represented by two values such as, for example, an x-axis valueand a y-axis value. The values may be expressed, for example, in pixels.

FIG. 2B shows the interface 202 and the cube 204 in world-space from anoblique angle such that it is clear that the view shown in FIG. 2Acorresponds to a projection of the cube 204 onto a window or a cameraimage plane of the 2D display interface 202. Visible again are the touchpoints 206 a-210 a and the cube 204. Also visible are a collection ofnormal projection lines 212 a-212 c that, for purposes of illustration,extend perpendicular from the plane of the interface 202 into aworld-space, until the projection lines 212 a-212 c intersect the cube204. The points on the cube 204 at which the normal projection lines 212a-212 c encounter the cube 204 correspond to a contact point 202 (notvisible in this view), a contact point 222 (not visible in this view),and a contact point 224. As such, the contact points 220, 222 and 224underlie the touch points 206 a, 208 a and 210 a, respectively, from theuser's point of view, as shown in FIG. 2A. In other words, thescreen-space location of a contact point 220, 222, 224 (i.e., thelocation of the contact point 220, 222, 224 when projected onto theplane of the interface 202) is equal to the screen-space location of itsrespective touch point 206 a, 208 a, 210 a. As shown in FIG. 2B,world-space is distinct from screen-space. The described techniques fordirect manipulation of a 3D object further distinguish betweenworld-space and object-space.

Object-space is a 3D space used to represent locations on a 3D objectrelative to an origin affixed to the 3D object such that the originmoves with the 3D object. Object-space locations can be expressed bythree values that represent, for example, displacements from the objectorigin along an x-axis, a y-axis, and a z-axis. Notably, a location of apoint on a 3D object expressed in object-space does not change withmovement of the 3D object.

World-space is a 3D space used to represent the location of a 3D objectrelative to a fixed origin. Notably, a location of a point on a 3Dobject that is expressed in world-space changes as the 3D object movesrelative to the fixed origin. The 2D display interface can be thought ofas a display of a view seen through a camera or a window of a world inwhich the 3D object is positioned. The image of the 3D object displayedby the 2D interface corresponds to a projection of the 3D object inworld-space onto the image plane of the camera or the window. Locationsin world-space can be expressed by three values that representdisplacements from the fixed origin. A point on an object inobject-space can be transformed into a point in world-space when theposition and rotational orientation of the 3D object in world-space areknown. After the point is transformed into world-space, the point canthen be transformed into screen-space by taking a projection of thepoint onto the image plane of the camera or the window.

For clarity and ease of exposition, a point at which the user “touches”the 2D display interface is hereinafter referred to as a “touch point”and is positioned in screen-space. The corresponding point on the 3Dobject that the user appears to “touch” because it is displayedunderneath the touch point is hereinafter referred to as a “contactpoint” and is positioned in object-space.

Referring now to FIGS. 2C and 2D, the user has moved his or her fingersto trigger an example transformation of the cube 204 by the interfacedevice (e.g., by sliding his fingers across the display interface 202while maintaining a continuous or a substantially continuous touching ofthe display interface 202). As shown in the user-perspective view ofFIG. 2C, the user moved his or her fingers in a rightward direction,across a portion of the interface 202. Movement of the user's fingersresulted in the movement of touch points 206 a, 208 a, and 210 a fromtheir position displayed in FIGS. 2A and 2B to their new positions 206b, 208 b, and 210 b, respectively. As seen in the oblique world-spaceview of FIG. 2D, when the user's fingers moved to the new touch pointlocations 206 b, 208 b and 210 b, the 2D display interface devicetransformed the cube 204 to ensure that the contact points 220, 222 and224 remain displayed underneath the same respective touch points 206 a,208 a and 210 a.

In general, the manipulation of the simulated 3D object is achieved bytransforming the 3D object (i.e., changing its location and rotationalorientation in world-space) when the user moves touch points from afirst location to a second location on the 2D display interface.Specifically, the location, rotational orientation and sometimes scalingof the 3D object in world-space is changed in response to the usermoving touch points such that the contact points on the object thatappear to be touched by the user remain displayed substantiallyunderneath the same touch points as those touch points are moved by theuser from one location to another location on the 2D display interface.

For example, when a user uses his/her fingers to place and move touchpoints, the manipulation of the 3D object is achieved by transformingthe 3D object such that the same contact points displayed by the pixelsunderneath a fingertip at the first location is then displayed by thepixels underneath the same fingertip at the second location when thefingertip moves from the first location to the second location. Thistransformation may involve, for example, solving fro transform valuesthat meet screen-space constraints. That is, the user placement andmovement of touch points controls the object's projection onto thecamera's image plane. Notably, while FIGS. 1 and 2A-2F (and other FIGS.in this document) illustrate 2D display interfaces in which a user useshis/her fingers to place and move touch points, other implementationsmay differ with respect to input mechanism. For example, in anotherimplementation, a stylus and/or other pointer mechanism may be usedinstead of or in addition to fingers to place and move touch points.

Referring back to FIGS. 2C and 2D, the cube 204 is transformed (e.g., byrotating and translating the cube) in world-space in such a way that thecontact points 220-224, when projected onto the image plane or window ofdisplay interface 202, are kept substantially underneath the new touchpoint locations 206 b, 208 b, and 210 b (as shown in FIG. 2D by normalprojection lines 212 a-212 c). That is, the cube 204 is transformed inworld-space such that the screen-space location of a contact point 220,222, 224 is substantially equal to the new screen-space location 206 b,208 b, 210 b of its respective touch point 206 a, 208 a 210 a.

As shown in FIGS. 2E and 2F, the user continues the rightward movementof his or her fingers across the 2D surface of the interface 202,thereby triggering a nearly 90-degree rotation of the cube 204 from itsoriginal position shown in FIGS. 2A and 2B. The user's fingers nowcontact the interface 202 at a collection of new touch point locations206 c, 208 c, and 210 c, and the cube 204 has been further transformed(e.g., rotated) in world-space to keep the projection of the contactpoints 220-224 onto the image plane or window of display interface 202substantially underneath the new touch point locations 206 c, 208 c, and210 c. Specifically, the cube 204 has been transformed in world-spacesuch that the screen-space location of a contact point 220, 222, 224 issubstantially equal to the new screen-space location 206 c, 208 c, 210 cof its respective touch point 206 a, 208 a, and 210 a.

FIG. 3 is a flow diagram of an example process 300 for moving simulated3D objects in response to one or more inputs on a substantially 2Ddisplay interface. In some implementations, the process 300 may be theprocess used to transform (i.e., rotate, translate and/or scale) the 3Dobjects of FIGS. 1 and 2A-2F. The process 300 may be implemented by a 2Ddisplay interface device such as, for example, a multi-touch displaydevice.

A determination (310) is made whether one or more new touch points havebeen added by the user. If the user has not yet interacted with the 2Ddisplay interface, then this determination will detect the user touchingthe screen for the first time (i.e., the user adding anew touch pointwhere before no touch points existed). If the user has already placed atleast one touch point on the 2D display interface, then thisdetermination will detect the addition of a new touch point. Theaddition of a new touch point may, for example, correspond to the userchanging from touching the displayed object with two fingers (i.e., twotouch points) to touching the displayed object with three fingers (i.e.,three touch points).

While operation 310 detects the placement of new touch points on thescreen, operation 320 detects movement of touch points already placed onthe screen. If no touch points are detected as being moved (320), thenthe process 300 returns to operation 310, and the 2D display interfacedevice continues monitoring for the addition of new touch points. Insituations where there are no touch points to move (e.g., the user isnot touching the screen), the process 300 still functions in that the 2Ddisplay interface device will simply not detect movement of touch pointswhen no touch points are available to be moved. The 2D display interfacedevice may, for example, monitor for changes in the status quo of touchpoint placement and movement by cycling between operations 310 and 320until either one or more touch points are moved or one or more new touchpoints are added to the set of touch points currently placed on thescreen.

If, however, one or more touch points have been determined (310) to havebeen added, then the new point or points the user has touched on thedisplay are identified (330). For example, the process 300 may identify(330) the touch points 206 a, 208 a, and 210 a of FIGS. 2A and 2B uponthem being newly placed on the 2D display interface 202 by the user.Identification of touch points 206 a, 208 a, and 210 a may include usingthe multi-touch display technologies to detect proximity of a fingerand/or a stylus tip to a portion of the 2D display interface 202 andthen identifying, for each detected portion, a corresponding location inscreen-space (e.g., for each touch point, an x axis and a y axisdisplacement expressed in pixels may be identified).

A contact point on the 3D object is also identified for each newly addedtouch point (340). For example, the contact points 220, 222 and 224 ofFIGS. 2B, 2D, and 2E may be identified as corresponding to the touchpoints 206 a, 208 a and 210 a, respectively. Stated differently,identifying a contact point for each newly added touch point includesidentifying a point on the 3D object in object-space that has ascreen-space location (i.e., a location of the projection of the pointon the 3D object onto the image plane of the camera in world-space) thatis the same as or substantially the same as that of the newly addedtouch point. A screen-space location of a contact point is alsosometimes referred to herein as a screen-space projection of the contactpoint.

In some implementations, if the newly added touch point is notpositioned on the display of the 3D object in the 2D display interface,the touch point may be disregarded with respect to transforming the 3Dobject since the user has effectively “missed” the 3D object with histouch, and, thus, movement of that touch point does not causecorresponding movement of the 3D object. In this case, no correspondingcontact point exists for the touch point. However, if the touch point issubsequently moved to a location over the display of the 3D object, acorresponding contact point can then be found and further movement ofthe touch point can result in movement of the 3D object viatransformation of the 3D object as described below. In otherimplementations, the 3D object may be transformed to ensure that anytouch point placed on the 2D display interface, regardless of itslocation, will contact the 3D object and have a corresponding contactpoint.

If movement of one or more touch points is detected (320), then the newscreen-space locations of the touch points on the display are identified(350). For example, if the user has moved the touch points 206 a, 208 a,210 a from the locations shown in FIG. 2A to the new locations 206 b,208 b, 210 b shown in FIG. 2B, then the new locations 206 b, 208 b and210 b are identified in screen-space.

A transformation of the 3D object is determined (360) to minimize theerror between the screen-space locations of the object contact pointsand the screen-space locations of their respective touch points on thedisplay. In other words, the amount of rotation and translation (and,depending on the implementation, scaling) of the 3D object inworld-space is determined such that the screen-space locations of theobject contact points are as close as possible to the screen-spacelocations of the touch points, which include at least one touch pointthat has just been moved to a new location. Any appreciable differencebetween the screen-space location of the object contact points and thescreen-space location of the touch points is deemed an “error” in thatthe object will appear to have slipped such that the object contactpoints are no longer displayed directly underneath the touch points. Agoal of this operation is to identify a transformation of the 3D objectthat eliminates this error or minimizes this error to within anacceptable range.

Notably, the set of touch points used for determining the transformationof the 3D object includes only those touch points that are currentlyplaced on the 2D display interface. That is, the 2D display interfacedevice detects whether any touch points have been removed prior todetermining the transformation of the 3D object and then determines thetransformation of the 3D object based only on the remaining touch pointscurrently placed on the 2D display interface.

The process 300 triggers transformation of the 3D object in response tomovement of touch points. Other implementations of the 2D displayinterface device may additionally trigger the transformation of the 3Dobject in response to adding or removing touch points or in response toapplying a different pressure to one or more existing touch points.Alternatively, the 2D display interface device may continually (orperiodically) transform the 3D object at fixed time increments (e.g.,every 1 millisecond), irrespective of movement, addition, or removal oftouch points or of pressure changes at touch points.

For example, in some implementations, the 2D display device mayadditionally detect when a touch point has been removed and, if thecurrent orientation of the 3D object corresponds to an error greaterthan zero (or greater than a predetermined small threshold), the 2Ddisplay device may immediately determine a new transformation of the 3Dobject that uses the remaining touch points to minimize the error andthen may transform and visually render the 3D object using the newtransformation. In this implementation, the removal of a touch point bythe user may appear to cause a 3D object that has slipped to suddenlysnap back (or snap closer) to the touch points that remain on thescreen.

In some implementations, the 2D display device may additionally triggertransformation of the 3D object in response to detecting that the userhas added a touch point that “missed” the 3D object. Upon detecting theuser adding such a touch point, the 2D display device may respond to thenewly added touch point by immediately transforming 3D object in orderto position a predetermined portion of the 3D object underneath thenewly added touch point.

In some implementations, the 2D display device may additionally triggertransformation of the 3D object in response to detecting that the userhas changed the pressure applied to one or more touch points, even ifthose points have not moved. As described further below in reference toambiguous rotation situations, application of pressure can set a targetdepth for the 3D object that can be used by the 2D display interfacedevice when transforming the 3D object. Accordingly, rather than movingtouch points across the 2D display interface, the user may insteadchoose to simply “push” one or more touch points to triggertransformation of the 3D object and watch the displayed 3D object appearto be pushed into the screen away from the user along the camera's Zaxis.

In some implementations, a function s(x,q) may be used to represent thescreen-space location of a contact point on an object. Specifically, thefunction s(x,q) defines the screen-space point p corresponding to anobject-space point x projected onto the image plane of the camera of the2D display interface when the object has been transformed in world-spacein accordance with a transformation vector q. The function s(x,q) may beexpressed as follows:

p=s(x,q)=h(PM(q)x)

In which P is a projection matrix, and M is a matrix parameterized bythe vector q which maps x from object-space into world-space, and h isthe viewport transformation. M is typically the product of severalmatrices which are parameterized by transform values that reflect howthe 3D object is moved world-space (e.g., through rotation, scaling,translation, etc.), and P is a projection matrix which describes thecamera's field of view as well as how an object's image size changeswith distance to the camera.

Typically, transforming a 3D object from object-space to world-spacerequires specification of values for six variables—three variables thatindicate the translation of the object along the x, y, and z axes andthree variables that indicate the rotation of the object about the x, yand z axes. Accordingly, a 3D object able to move freely in world-spacehas six degrees of freedom (DOFs). When projecting this 3D object ontothe image plane of a camera, an additional DOF corresponding to objectscaling (i.e., how the object's image size changes with distance fromthe camera) also may be be considered. Accordingly, a transform of a 3Dobject from object-space to world-space when considering the projectionof the 3D object onto an image plane of a camera may involve seventransform DOFs.

Depending on the choice of the projection matrix P, however, some ofthese DOFs may be redundant or unnecessary. For example, if P is chosento be a perspective transformation, scaling the object up may be similarin effect to translating the object so it becomes closer to the camera.However, if P is chosen to be an orthographic projection, thentranslating the object toward or away from the camera may have noeffect, perhaps leaving scaling as the sole DOF capable of changing theobject's screen-space extent. For these reasons, in someimplementations, not all seven transform DOFs (three for rotation, threefor translation, and one for scaling) may be required or even useful atall times.

In some implementations, to simplify the transformation calculations,the camera is assumed to be a perspective camera and thus an object'sscreen-space extent is affected by both translation and scaling.However, in recognition that, when viewed from a perspective, camera, anobject translating along the camera's Z axis is visually identical tothe scaling of the object, the scaling DOF is removed from thetransformation calculations, resulting in the object having only 6 DOFs.Additionally, the projection matrix P is made constant to preventintroduction of any other DOFs arising from movement of the camera. Assuch, the projection of an object contact point onto the image plane ofthe camera may only be altered by the object-space to world-spacetransformation as controlled by the transformation vector q because thelocation of the camera is kept fixed (i.e., the camera location has notDOFs).

The transform parameters q and object-space to world-spacetransformation matrix M(q) may be defined as:

q=[tx ty tz qx qy qz]

M(q)=T(tx,ty,tz)Q(qx,qy,qz)

in which T is a conventional translation matrix and Q is aunit-quaternion matrix in which:

qw=√{square root over (1−qx ² −qy ² −qz ²)}

In this implementation, quaternions are used to represent rotations ofthe 3D object. Quaternions are normally a four DOF quantity, but if thetransformation vector q is forced to contain only unit quaternions(e.g., by constraining the value of qw as shown above) then the numberof quaternion DOFs can be dropped down to three. Accordingly, in thisimplementation, identifying a 3D transformation of an object to minimizethe error between the screen-space location of the object contact pointsand the new screen-space location of the touch points requires adetermination of six transform parameters (tx, ty, tz, qx, qy and qz) ofthe vector q.

In some implementations the 2D display interface device includes asolver that determines the transform parameters of the transformationvector q by minimizing a quadratic energy function that measures thetotal squared error between the screen-space locations of the touchpoints (including the newly moved touch points) and the screen-spacelocations of the corresponding contact points. For example, if s(x,q) isa function that projects object-space point x into screen-space giventransformation vector q, and x_(i) and p_(i) are the positions of thei-th contact point and i-th touch point, respectively, then the energy(i.e., total squared error) may be given by the following equation:

$E = {\sum\limits_{i}^{\mspace{11mu}}\; {{{s\left( {x_{i},q} \right)} - p_{i}}}^{2}}$

That is, E is equal to the square of the Euclidean distance between thescreen-space location of an object-space contact point x_(i) and ascreen-space location of its corresponding touch point p_(i) summed upfor all i, where i is equal to 1 to the total number of touch pointsplaced on the screen by the user. Values for the transformation vector qare then sought by the solver that minimize E with respect to q. Thisminimization may be treated as a nonlinear least-squares minimizationproblem. A variety of techniques can be used by the solver to solve forthe values of the transformation vector q that result in a minimum errorbetween the screen-space locations of the contact points on the 3Dobject and the screen-space locations of the touch points (one or moreof which have been moved to new locations). For example, the solver mayuse purely gradient-based algorithms. Additionally or alternatively, thesolver may use a Quasi-Newton method such as theBroyden-Fletcher-Goldfarb-Shanno (BFGS) method.

In one implementation, the solver uses a Levenberg-Marquardt nonlinearleast-squares minimization algorithm (LM) because of the speed at whichit can be executed on commodity hardware. For example, by using the LMalgorithm, the solver can solve for the transformation vector q in afraction of a millisecond on a 3 GHz Intel CPU. This speed ofcalculation enables a conventional personal computer to transform the 3Dobject in real-time such that movement of the touch points by the userresults in a corresponding real-time transformation and display of the3D object that creates a smooth and realistic illusion of objectmanipulation by the user.

Using the LM algorithm, however, may require at least as many terms inthe energy function E as the DOFs of the object. Since each touch pointadds two terms to the energy (i.e., an error in the x screen-spacecoordinate and an error in the y screen-space coordinate),transformations based on three touch points of an unconstrained objectshould result in a transformation vector q that transforms the objectwith no error or slippage (i.e., E=0). That is, since the number of DOFsof an unconstrained 3D object is six and the number of terms in theenergy function E for the three touch points is also six, the LMalgorithm should be able to solve for a transformation vector q thattransforms the 3D object such that the screen-space locations of thecontact points on the 3D object are equal to the new screen-spacelocations of the touch points (i.e., E=0).

In contrast, if only one or two touch points have been placed on thescreen, then use of the LM algorithm may require locking down some ofthe DOFs to the transform's current values. For example, if only onetouch point is used for the transformation (e.g., only one finger isplaced on the 2D display interface and is moved from one location to theother), then 4 DOFs (i.e., 6 DOFs-2 DOFs) may be fixed at thetransform's current values. For example, the values of tz, qx, qy and qzmay be fixed to the transform's current values such that movement of thefinger only results in the object translating in a plane defined by thex and y axes (i.e., movement of the object is limited to translation inthe x and y directions of the camera). If only two touch points are usedfor the transformation (e.g., only two fingers are placed on the 2Ddisplay interface and are moved to different locations), then 2 DOFs(i.e., 6 DOFs-2×2 DOFs) may be fixed at the transform's current values.For example, the values qx and qy may be constrained to fix therotational orientation of the object about the x and y axes such thatmovement of the two fingers only results in translation of the object inthree dimensions and rotation of the object about the z-axis. Notably,by locking down some of the DOFs at the transform's current values,application of the LM algorithm result in a transformation of the objectthat is entirely consistent with conventional RST-style 2D objecttransformation for one and two touch point interactions. Moreover, byconstraining the DOFs of the object as stated above for one to two touchpoint interactions, the object can be transformed with no error orslippage (i.e., E=0). That is, the transformation vector q of the objectfound by the LM algorithm should result in the screen-space locations ofthe contact points on the 3D object being equal to the new screen-spacelocations of the touch points.

In some implementations, an interaction with the 2D display interfacemay be deemed to begin upon placement of one or more touch points on theinterface and may end upon the user removing all touch points from theinterface. Alternatively, an interaction with the 2D display interfacemay be deemed to begin upon a user placing a set number of touch pointson the screen (e.g., one touch point placed by one finger) and end whenthe user later changes the number of touch points on the screen (e.g.,changes to two touch points placed by two fingers).

When four or more touch points are placed on the 2D display interface,use of an unconstrained energy quadratic in screen-space error mayenable the interface to continue to be responsive. However,minimizations which involve a large number of touch points may come atthe expense of some slippage. In some implementations, such slippage maybe spread out evenly among the touch points.

While the one to two touch point interactions described above involvedconstraints on the DOFs of the 3D object by fixing the DOFs at theircurrent transform values, in some implementations, other constraints mayplaced by the interface designer on the DOFs of the 3D object. Theseother constraints are referred to as penalties and are described furtherbelow with respect to FIGS. 14-19.

Referring again to FIG. 3, the 2D display interface device uses thedetermined transformation to transform the 3D object (370), and visuallyrender the transformed 3D object (380) to make the displayed objectcontact points substantially underlie the location of their respectivetouch points on the 2D display interface (including the one or morenewly moved touch points). For example, in FIG. 2F, the 2D displayinterface device rotated the cube 204 to place the screen-spaceprojection of the contact points 220-224 underneath the new screen-spacelocation 206 c, 208 c, 210 c of touch points 206 a, 208 a and 210 a,respectively.

Notably, if the transformation determined by the solver in operation 350corresponded to no error or slippage (e.g., E=0 for the transformationvector q found by the solver), then the screen-space locations of thecontact points should be identical to that of their respective touchpoints. That is, a displayed contact point (i.e., the projected contactpoint) may be exactly underneath its corresponding touch point withouteven one pixel of deviation. If the transformation determined inoperation 350, however, corresponded to some error or slippage (e.g.,E>0 for the transformation vector q found by the solver), then thescreen-space locations of the contact points will differ from that oftheir respective touch points. That is, a displayed contact point maynot be positioned exactly underneath its corresponding touch point butrather may be separated, for example, by one or more pixels from itscorresponding touch point. FIGS. 5A-5E illustrate an example of slippageor error between touch points and their corresponding contact points andare described in more detail below.

Once the object has been transformed and rendered, the determination(310) is made again and the process 300 continues. In oneimplementation, the 2D display interface device repeats the process 300every 1 millisecond such that the placement and movement of touch pointson the 2D display interface is continually monitored and objects aretransformed and rendered by the 2D display interface device virtuallyimmediately in response to touch point changes on the 2D displayinterface.

The process 300 supports direct manipulation for substantially anynumber of touch points in both 2D and 3D. In examples in whichinteractions are based on a single touch point or on two touch points,process 300 provides output that is consistent with that of a standardRST controller for 2D objects by fixing some of the DOFs of the objectto the transform's current values. That is, the process 300 outputs thetransform values that result in the screen-space projections of thecontact points matching the screen-space locations of the touch pointsafter the movement of one or more of the touch points by the user to newlocations on the 2D display interface.

With three or more touch points, the process 300 supports directmanipulation into 3D. As is the case with one or two touch points in 2D,the process 300 produces exact solutions in 3D for up to three touchpoints (i.e., solutions with no slippage that have E equal to zero orapproximately equal to zero). As described above, for four or more touchpoints, minimizing the energy equation will find the transformparameters resulting in the substantially best fit to screen-spacecontact point location. In some implementations, the process 300 mayprovide a user with a way to easily rotate and translate displayed 3Dobjects with little training because of the intuitive nature of 3Dobject direct manipulation.

The process 300 is described above in terms of manipulation of a 3Dobject as if the 3D object were a discrete unit and in terms of contactpoints located on the surface of the 3D object. In practice, however,the 2D display interface device that implements process 300 is operatingon data sets that represent the 3D object.

For example, when transforming and rendering a 3D object on the 2Ddisplay interface, the 2D display interface device may access a 3Dobject-space data set that represents the 3D object in object-space(e.g., the data set identifies every point in object-space thatcorresponds to a location on the surface of the object and/or within theobject). The 2D display interface device also may access datarepresenting an object transformation vector that specifies theorientation of the 3D object in world-space (i.e. specifies therotational orientation, location, and, in some implementations, scalingof the object). Using the 3D object-space data and the transformationvector data, the 2D display interface device may calculate a 3Dworld-space data set that represents the 3D object in world-space (e.g.,the data set identifies every point in world-space that corresponds to alocation on the surface of the object and/or within the object). The 2Ddisplay interface device may determine the 3D world-space data set by,for example, determining an object-space to world-space conversionmatrix for the transformation vector and then applying the matrix to the3D object-space data set. The 2D display interface device may thenaccess data representing an image plane of a camera of the 2D displayinterface and may then determine a 2D screen-space data set for theobject that represents a 3D view of the 3D object by projecting the 3Dworld-space data set onto the image plane based on the data representingthe image plane. The 2D display interface device may then render the 3Dview of the 3D object using the 2D screen-space data set.

Referring now to FIGS. 4A, 4B, 4C and 4D, the techniques describedherein enable a variety of bimanual (e.g., two-handed) interactions formanipulating displayed 3D objects. Perhaps the most natural of these isa swivel interaction, an example of which is illustrated in FIG. 4A. Inthis interaction, the user places two (or more) fingers of a hand 412down on a 3D object 410, which in some implementations may pin theobject 410 down at these touch points and defines an axis 416 to swivelaround. Then, by moving one finger from the other hand 414 while keepingthe fingers of hand 412 fixed, the user can freely swivel the object 410about.

In reality, the fingers of hand 412 pinning the object 410 down do notdefine a fixed axis. While the user can rotate the object 410 as ifthere really was an axis defined, the user is also able to manipulatethe object 410 in directions parallel to the axis 416. An example ofthis type of interaction is shown in FIG. 4B. In some implementations,this type of manipulation may not be possible if the axis of rotationwere fixed. In comparison, a “handle controller”, which defines a fixedaxis with one hand and controls the angle of rotation with the other,may be more cumbersome compared to the type of interaction the describedtechniques permit. In some implementations, the described techniques maybe extended to mimic a handle controller or other indirect 3Dmanipulators by placing penalties to impose screen-space constraintsthat mimic those imposed by a handle controller. Using penalties toimpose screen-space constraints is described further below with respectto FIGS. 14-19.

FIG. 4C illustrates another example three-finger interaction, which maybe called a “shear rotate”. As in the swivel interaction illustrated inFIG. 4A, the user places three fingers on the object 410, two from thehand 414 and one from the other hand 412. The user then rotates the hand414 that has two fingers down through an angle, in this example, byapproximately 90 degrees. As the three points become more collinear, theobject 410 rotates either towards or away from the user, depending onthe object's 410 orientation at the beginning of the interaction. Ineither case the result may be that the object 410 will have rotatedabout 90 degrees about an axis in the camera's X-Y plane, while twistingapproximately 45 degrees about a perpendicular axis also in camera's X-Yplane.

Referring now to FIG. 4D, illustrated is an example interaction whichmay be referred to as a “perspective rotate”. In this interaction, theuser begins by placing four fingers at a collection of touch points420-426 corresponding to contact points on the object 410 in a roughlyrectangular configuration. The user then moves the touch points todecreases the distance between two adjacent touch points, such as, forexample, shrinking the distance between the touch points 420 and 422 andthe touch points 424 and 426, while simultaneously increasing thedistance between the other two, such as increasing the distance betweenthe touch points 420 and 424 as well as between the touch points 422 and426. In response to this gesture, the object 410 may rotate to bestachieve the perspective described by the new hand configuration.

FIGS. 5A-5E illustrate an example of an interaction with a displayed 3Dobject that results in slippage. As stated above, if four or more touchpoints are placed on the 2D display interface, the energy function E maybe overconstrained such that the solver attempting to find the 3D objecttransformation may fail to converge, and the display may becomeunresponsive (e.g., appear to lock up because the computer is busytrying to resolve a combination of inputs for which no exact solution isachievable). However, by using an unconstrained quadratic minimization,the solver may find an acceptable 3D object transformation that has someerror or slippage (i.e., has some deviation in screen-space contactpoint locations from the new screen-space locations of the correspondingtouch points such that E>0) but that distributes the error or slippageamong the touch points. In such an implementation, any number of touchpoints may be placed, even if the total number of input DOFs (whichcorrespond to two input DOFs per touch point in the 2D plane of the 2Ddisplay interface) is larger than the number of transform parametersbeing solved for. Moreover, as described further below with respect toFIGS. 12A, 12B, 12C and 13, the error or slippage can be distributedamong the touch points in a non-uniform manner that may, for example,place the majority of the slippage on one or more of the touch points.

Referring now to the user perspective view of FIG. 5A, a 3D object 502is displayed on a 2D display interface 504. The user has placed twofingers on one hand at a touch point 506 a and a touch point 508 a, andhas placed two fingers from the other hand at a touch point 510 a and atouch point 512 a. As can be seen in the oblique world-space view shownin FIG. 5B, the touch points 506 a, 508 a, 510 a and 512 a correspond toa collection of contact points 520, 522, 524 and 526 on the object 502.

FIG. 5C and 5D illustrate the user and oblique world-space views of theobject 502 and the interface 504 after the user has rotated his/herright hand while keeping the position of his/her left hand fixed. Assuch, the locations of the touch points 506 a and 508 a remainsubstantially unchanged, but the rotated right hand has moved touchpoints 510 a and 512 a across the surface of the interface 504 to newlocations 510 b and 512 b, respectively.

FIG. 5E is an enlarged view of the area identified as 550. As can beseen, the new locations 510 b and 512 b are offset from the originallocations of touch points 510 a and 512 a, while the locations of touchpoints 506 a and 508 a remain substantially unchanged. This illustratesan example of the previously described scenario where the movement offour or more touch points may make an exact minimization (e.g., E=0)impossible. In other words, no combination of transform parameters qexists that would orient the object 502 in world-space in a way that canalign the contact points 520, 522, 524 and 526 with the touch points 506a, 508 a, 510 b, and 512 b such that the screen-space locations of eachof the contact points 520, 522, 524 and 526 is equal or substantiallyequal to the screen-space locations of their respective touch points 506a, 508 a, 510 b and 512 b. The discrepancy or deviation between thescreen-space locations of the touch points 506 a, 508 a, 510 b, and 512b and the screen-space locations of the contact points 520, 522, 524 and526 is an example of the aforementioned phenomena of slippage.

As shown in FIG. 5E, a solution for q produced by the energyminimization algorithm (e.g., the LM algorithm) results in the 2Ddisplay interface device slightly rotating the object 502counterclockwise relative to the plane of the interface 504 tosubstantially evenly distribute the error among the touch points. Thetouch points 506 a, 508 a, 510 b, and 512 b are visible, along with acollection of screen-space projection locations 562, 564, 566 and 568 ofthe contact points 520, 522, 524 and 526. Notably, in this example,since the object 502 cannot be transformed in such a way as to cause allfour of the screen-space locations 652-568 to perfectly align with thescreen-space locations 506 a, 508 a, 510 b, and 512 b (which may, forexample, occur when a solution for the transform parameters q that makesE=0 cannot be found), a solution for q is found that minimizes the errorbetween the screen-space locations 562, 564, 566 and 568 of the contactpoints 520, 522, 524 and 526 and the new screen-space location of thetouch points 506 a, 508 a, 510 b, and 512 b (e.g., minimizes the valueof E).

User manipulations of simulated 3D objects by using the described 2Ddisplay interface device may produce some surprising results thatnegatively impact the desired illusion that the user is actuallymanipulating a real 3D object. For example, some user manipulations canlead to surprising results in which the 3D object suddenly rotates aboutaxes other than the camera's Z axis in a manner that was unexpected bythe user. Such surprising results may be caused by rotationalextrema—i.e., points in the transform space at which screen-spacedistances are maximized with respect to rotational DOF. These situationsmay arise frequently during normal use.

Two major classes of surprising results attributable to rotationalextrema are identified and techniques for curbing their negativeinfluence are provided. The fact class is referred to as ambiguousrotations and is described below in references FIGS. 6A, 6B, and 7, andthe second class is referred to as rotational exhaustion and isdescribed below in reference to FIGS. 8( i)-8(v), 9(i)-9(v), 10A, 10Band 11.

Ambiguous rotation situations occur when three or more touch pointsapplied by the user have corresponding contact points that are locatedat the same depth along the camera's Z axis and the user then brings thethree touch points closer together such that the 3D object will have arotate about an axis to match the touch points with the contact points.Such a situation is shown in FIGS. 6A and 6B in which the user hasapplied a collection of three touch points 606, 608, and 610 thatcorrespond to contact points 620, 622, and 624, which are located atapproximately the same depth along the camera's Z axis. An ambiguousrotation results, for example, when the user keeps the location of twoof the three touch points fixed and then moves the third touch pointtowards the two fixed touch points. In moving the third touch pointtowards the two fixed touch points, the user may expect the object torotate about an axis lying in the X-Y plane of the camera that isdefined by the two fixed touch points. The user, however, may besurprised when the 2D display interface shows the object rotating aboutthe expected axis but in a direction opposite to what the user intended(i.e., the object is rotated counterclockwise about the expected axisrather than clockwise or vice versa).

FIGS. 7( a), 7(b) and 7(c) are diagrams that illustrate an example of anambiguous rotation of a 3D object 710 displayed on a 2D displayinterface 705. The initial locations of three touch points 714, 716 and718 are shown in FIG. 7( a). The three touch points 714, 716 and 718correspond to contact points on the 3D object 710 located at the samedepth along the Z-axis of the camera. The user keeps the locations ofthe two touch points 714 and 716 fixed on the 2D display interface 705so that they define an axis of rotation 712. As shown by the arrow in inFIG. 7( a), the user then moves the third touch points 718 closer to theaxis of rotation 712.

FIG. 7( b) shows a first of two possible displays that may be renderedon the 2D display interface 705 in response to the user moving the thirdtouch point 718 closer to the axis of rotation 712. As shown in FIG. 7(b), the 2D display interface 705 displays the object 710 rotated awayfrom the user about the axis of rotation 712. Notably, thetransformation of the object 710 produced in response to the detectedmovement of the third touch point 718 exhibits no slippage (i.e., E=0,such that the screen-space location of each contact point issubstantially equal to the screen-space location of its correspondingtouch point after transformation of the object).

FIG. 7( c) shows a second of the two possible displays that may berendered on the 2D display interface 705 in response to the user movingthe third touch point 718 closer to the axis of rotation 712. As shownin FIG. 7( c), the 2D display interface 705 displays the object 710rotated towards the user about the axis of rotation 712. Notably, likethe transformation of the object 710 shown in FIG. 7( b), thetransformation of the object 710 shown in FIG. 7( c) also exhibits noslippage (i.e., E=0, such that the screen-space location of each contactpoint is substantially equal to the screen-space location of itscorresponding touch point after transformation of the object).

Accordingly, as shown in FIGS. 7( b) and 7(c), two equally correcttransformation vectors q corresponding to opposite directions ofrotations about axis 712 can be found for transforming the object 710without slippage in response to movement of the third touch point 718towards the axis of rotation 712. Assuming that a gradient descent-basedsolver is used (i.e., a solver that uses a gradient descent-basedalgorithm such as, for example, the LM algorithm), the choice oftransformation vector q from among these two possible transformationvectors q may be determined entirely by how the solver chooses descentdirections.

However, by intelligently biasing the starting conditions used by thesolver when applying the gradient descent-based algorithm to favor onedirection over the other, it is possible to choose which of the twopossible transformation vectors q will be selected by the solver.Specifically, a situation in which an ambiguous rotation is possiblecorresponds to a maxima in the solution space of the solver. By applyinga small perturbation to the initial values of the solver away from themaxima, a particular descent path in the solution space be be chosen atthe start that will arrive at one solution for the transformation vectorq over the other. For ambiguous rotations, this may involve selectingthe axis of rotation and a direction of rotation about that axis andthen using a small displacement in that direction as the starting pointfor rotational update DOF of the solver (e.g., quaternion values qx, qyand qz).

For example, in the situation presented in FIGS. 7( a), 7(b) and 7(c),the solver can be biased to select the transformation vector q thatrotates the object 710 towards the user by initializing transformationparameter qy to a small negative value (such as −10⁻¹) instead of zero.The solver will then start by considering solutions for thetransformation vector q that include a clockwise rotation of the object710 and the Y axis.

The biasing of the solver to select one transformation vector q over theother transformation vector q in an ambiguous rotation situation alsocan be based on a set of rules that take into account user interaction.In one implementation example, the set of rules may indicate that thestarting conditions of the solver should be biased to result in anegative rotation about the Y-axis when the movement of the third touchpoint closer to the two other touch points corresponds to moving thethird touch point left on the screen (i.e., on the 2D display interface)and result in a positive rotation about the Y-axis when the movement ofthe third touch point closer to the two other touch points correspondsto moving the third touch point right on the screen. Similarly, thestarting conditions of the solver may additionally or alternatively bebiased such that moving the third touch point upwards and downwards onthe screen results in a positive and a negative rotation about theX-axis, respectively. The biasing in accordance with these rules may bearrived at by biasing (or perturbing) the starting values of qx and/orqy such that descending the gradient in the solution space from thelocal maxima corresponding to the ambiguous rotation situation willcause selection of the transformation vector q that corresponds to thedesired rotation.

Notably, the biasing or perturbation of the starting conditions used bythe solver to mitigate ambiguous rotation situations should not impactthe ability of the solver to arrive at a correct energy minimizationtransformation vector q, even when the solver is not faced with anambiguous rotation situation. The correct energy minimizationtransformation vector q corresponds to a local minima in the solutionspace. The biasing or perturbation of the starting conditions used bythe solver to deal with ambiguous rotations will either push the DOF ofthe object towards or away from the local minima in the solution space.If the initial starting value is perturbed towards the local minima,then the solver will converge to the correct energy minimization vectorq as normal. If the initial starting value is perturbed away from thelocal minima, then, because the starting point is away from the localmaxima (i.e., the solver is not faced with an ambiguous rotationsituation), and assuming the perturbation is relatively small, thesolver should still converge to the correct energy minimization vectorq. That is, the solver will descend the gradient from the perturbedpoint past the original pre-perturbed point and wind up at the samelocal minima corresponding to the correct energy minimization vector q.

In some implementations, pressure also may be used in addition to orinstead of biasing the solver to correct rotational ambiguities. Inimplementations wherein the 2D display interface is pressure sensitive,the user may vary the pressure applied at the different touch points toinfluence the transformation of the 3D object. The application ofpressure by the user when placing and moving touch points may providethe user with an extra half-dimension (e.g., a non-negative value) thatcan be used by the 2D display interface device in addition to the usual2D touch point placement and motion for determining the transformationof the 3D object. The application of pressure by the user, thus, mayenable the user to “push” a contact point further into the screen.Rotational ambiguities then may be resolved by pushing one side of theobject into the screen when beginning a rotation.

In some implementations, pushing can be accomplished by adding a penaltyto the energy equation E which influences the solver into moving acontact point (that corresponds to the touch point at which thepressured was applied) in the desired direction in world-space. Thedesired depth, z_(target), can be determined from the pressure valueapplied at the touch point and the corresponding contact point's currentworld-space depth. For example, if the contact point's currentworld-space depth is 1200 in world-space units and the pressure appliedby the user at the touch point corresponding to the contact point is P,the target world-space depth for the contact point may be 1200 plusP/10. The contact point's deviation from the target depth can then bemeasured via the quadratic penalty

λ(z _(target) −<z, M(q)x>)²

in which x is the contact point in object-space to be pushed, z is theunit-length vector pointing into the screen, and λ is a weightingcoefficient used to correct the difference in dimensionality between theworld-space penalty and the screen-space energy. The term M(q)xrepresent the contact point expressed in world-space and is equal to thecontact point x in object-space multiplied by the object-space toworld-space transformation matrix M(q). The term <z, M(q)x> representsthe inner product (i.e., the dot product) of the unit-length vector zand the contact point expressed in world-space, which is simply the zcoordinate i world-space of the contact point. Accordingly, the penaltyapplied to the energy equation E is equal to λ times the square of aquantity equal to the difference between z_(target) and the z coordinatein world-space of the contact point x.

The term λ can be viewed as a trade-off factor of how much slippageamong the contact points is acceptable for the point pushed into thescreen. A more in depth discussion of penalties and possible choices ofvalue for λ is provided below in reference to FIGS. 14-19.

In some implementations, a pressure penalty term is included in theenergy equation E for each contact point corresponding to a touch pointplace don the screen (i.e., on the 2D display interface). In otherimplementations, a pressure penalty term is only included in the energyequation E for each of the contact points corresponding to touch pointsplaced on the screen by the user and for which the user has applied apressure greater than a predetermined threshold value.

As stated previously, another class of surprising results attributableto rotational extrema that occur in response to user manipulations ofsimulated 3D objects using the described 2D display interface device isrotational exhaustion. Unlike ambiguous rotation situations which arisewhen three or more touch points are moved closer together, rotationalexhaustion situations arise when three or more touch points are movedfarther apart.

FIGS. 8( i), 8(ii), 8(iii), 8(iv) and 8(v) are diagrams showing anexample of a rotational exhaustion situation. As shown in FIG. 8( i),the user has placed three touch points 812, 814 and 816 on the 2Ddisplay interface. The three touch points 812, 814 and 816 havecorresponding contact points B, A and C, respectively, such that the 2Ddisplay interface creates the illusion that the user is grasping theobject 810 at the corresponding contact points. Note that in thisexample no slippage occurs, and therefore, the contact points, whilelabeled A, B and C in FIG. 8, are not separately visible underneath thecorresponding touch points 814, 812 and 816.

The user then moves the touch point 812 to the right on the interfacewhile keeping the location of the touch points 814 and 816 fixed. Inresponse to the movement, the 2D display interface device rotates theobject 810 about the axis 820 such that the contact points A, B and C onthe object 810 remain substantially underneath the touch points as shownin FIG. 8( ii).

The user continues to move the touch point 812 to the right on the 2Ddisplay interface while keeping the touch points 814 and 816 fixed untilthe object 810 has exhausted its rotation about the axis 820, as shownin FIG. 8( iii). Put another way, the user continues to move the touchpoint 812 to the right on the 2D display interface until thescreen-space lengths between the contact point B and the respectivecontact points A and C are maximized with respect to rotation about theobject's current axis, i.e., rotation about axis 820.

When rotation about the current axis (i.e., axis 820) of the object 810is exhausted, further movement of the touch point 812 to the right onthe interface may still result in the solver determining atransformation of the object that corresponds to zero slippage by usingother object DOFs to move the object. However, the sudden shift to usingthe other object DOFs when matching the touch points with the contactpoints can result in movement of the 3D object in a surprising andunintuitive manner.

As shown in FIG. 8( iv), the continued movement of touch point 812 tothe right on the 2D display interface after the rotation about the axis820 of the object 810 has been exhausted results in the 2D displayinterface device both translating the object 810 in the Z-axis directionto bring the object 810 closer to the image plane of the camera androtating the object 810 about a new axis 822 that is perceived to beperpendicular to the axis 820. The rotation about the new axis 822occurs because of the constraints imposed on the movement of the object810 by contact points A and C as the object translates in the Z-axisdirection to accommodate the continued movement of the touch point 812to the right of the 2D display interface. Further movement of touchpoint 812 to the right on the 2D display interface results in the object810 moving even further towards the image plane of the camera and theobject 810 rotating even more about the new axis 822, as shown in FIG.8( v).

Many users of the 2D display interface may find the sudden and abrupttranslation of the object 810 toward the camera and the simultaneoussudden and abrupt change in the rotational axis of the object 810 to beinconsistent with the original interaction, which was merely movement ofa single touch point (i.e., touch point 812) in a straight line in onedirection on the 2D display interface. More generally, users may expectthat a smooth and consistent motion of touch points on the 2D displayinterface should result in an equally smooth and consistent motion ofthe object similar to that which results from a 4 DOF interaction whenmanipulating a 2D object. As shown above, however, this property is notsatisfied under projected 3D rotation when rotational exhaustion occurs.That is, as an axis of rotation is exhausted, sharp changes in directionin parameter space may occur. These sudden and unexpected changes inmovement of the object may be surprising to many users and may causeconfusion. As described next, the point at which the rotationalexhaustion occurs may be detected and appropriate action may be taken toeither control or correct the unexpected shift in interaction behavior.

Some insight into the problem of rotational exhaustion may be gained byfitting a plane to the contact points A, B and C corresponding to touchpoints 814, 812 and 816 and examining the path the plane's normal takesduring a typical interaction exhibiting rotational exhaustion. FIGS. 8(i)-8(v) include a representation of the position of the normal to thebest-fit plane below each respective illustrated position of object 810.The representation of the position of the normal is a fixed hypotheticalsphere that encompasses the object 810 and on which is depicted thelocation that the normal intercepts the surface of the fixedhypothetical sphere as the normal moves with the movement of object 810.Specifically, the path that the normal has taken is depicted on thefixed hypothetical sphere by a line and the current position of thenormal is depicted on the fixed hypothetical sphere by a small circlethat represent the current location that the normal intercepts thesurface of the sphere. Notably, the position of the eye vector of thecamera (i.e., the unit vector that points in the camera's Z-axisdirection) is represented on the fixed hypothetical sphere as theclosest point on the sphere to the reader.

FIG. 8( i) shows the sphere 824 depicting the initial position of thenormal of the best-fit plane of the contact points A, B and C at thebeginning of the movement of object 810 that will lead to rotationalexhaustion. Notably, the best fit plane, in this example, is the planethat encompasses the flat side of the object 810 (i.e., the side of thecube) on which is positioned the three contact points A, B and C.

As the touch point 812 is moved by the user to the right on the 2Ddisplay interface, the 2D display interface device rotates object 810about the axis 820 and the normal of the of best-fit plane starts tobecome more and more parallel with the eye vector of the camera. This isdepicted by spheres 826 and 828 of FIGS. 8( ii) and 8(iii),respectively.

When the user has moved the touch point 812 to the right on the 2Ddisplay interface such that the rotation about the axis 820 isexhausted, the 2D display interface device suddenly translates towardsthe camera and begins rotating the object 810 about a new axis 822. Thisnew rotation causes the normal of the best-fit plane to suddenly take asharp turn and become less and less parallel with the eye vector of thecamera. This sharp turn is depicted by sphere 830 of FIG. 8( iv).Continued movement by the user of touch point 812 to the right on the 2Ddisplay interface results in the 2D display interface device furthertranslating the object toward the camera and further rotating the object810 about the new axis 822 such that the normal of the best fit planecontinues along its new trajectory, as shown by sphere 832 of FIG. 8(v).

Notably, the sharp bend in the path that the normal takes characterizesrotational exhaustion as the axis of rotation quickly changes from oneaxis to another. While FIGS. 8( i)-8(v) show the new axis 822 and theinitial axis 820 being substantially perpendicular to each other andshow the normal of the best-fit plane becoming substantially parallel tothe eye vector when rotational exhaustion occurs, such is not always thecase. In some rotational exhaustion situations, the new axis may not beperpendicular to the initial axis, and the normal of the best-fit planemay not ever become parallel to the eye vector. Moreover, while thesituation illustrated in FIGS. 8( i)-8(v), depicts both the new axis 822and the initial axis 820 lying on the X-Y plane, it is important to notethat rotational exhaustion can occur from any starting objectorientation, rotating about any axis, not only those axes which lie onthe X-Y plane.

Despite the different flavors of rotational exhaustion, rotationalexhaustion may still be dependably detected by tracking the path thenormal of the best-fit plane takes relative to the eye vector of thecamera. When the normal of the best-fit plane becomes more and moreparallel with the eye vector of the camera, the plane is said to bewaxing. When the normal of the best-fit plane becomes less and lessparallel with the eye vector of the camera, the plane is said to bewaning. Rotational exhaustion may occur at the plane's transition pointfrom waxing to waning. Put differently, rotational exhaustion may occurwhen the normal of the best-fit plane is maximally parallel to the eyevector and then starts turning away. Notably, the transition point fromwaxing to waning does not necessarily mean that the normal vector andthe eye vector are actually parallel, but rather that, given theconstraints set by the contact points, they may not get any moreparallel than they are at this point.

The transition point from waxing to waning can be detected by the 2Ddisplay interface device by monitoring the angle between the normal ofthe best-fit plane of the contact points and the eye vector. In someimplementations, w(x,q) is defined as a function which measures theangle point x's normal makes with the eye vector at x, given transformparameters q. The function is evaluated at the centroid of the contactpoints (i.e., x=x_(centroid)) using the normal of the best-fit plane.When the function is evaluated over times as the object orientationchanges (i.e., as the transform parameters q change over time), a localminimum in the function with respect to time corresponds to the best-fitplane transitioning from waxing to waning and, therefore, may beindicative of a rotational exhaustion situation. In particular, thefunction w may be expressed as follows:

w(x_(centroid,q))=1−<M(q)eye_(centroid), M(q)n_(plane)>

where x_(centroid) is equal to the location of the centroid of thecontact points in object-space, M(q) is the object-space to world-spacetransformation matrix, eye_(centroid) is the unit length camera eyevector expressed in object-space at the location of the centroid of thecontact points, and n_(plane) is the unit length normal to the best-fitplane. Accordingly, the value of the function w at the centroid of thecontact points is equal to one minus the quantity equal to the inner/dotproduct of the unit-length eye vector in world-space at the centroid(i.e., the vector that points from the centroid to the camera) and theunit length normal to the best-fit plane in world space. The value forw, therefore, may range from 0 (when the camera eye vector and thenormal vector are parallel) to 1 ( when the camera eye vector and thenormal vector are perpendicular to each other). As stated above, localminimums in w correspond to transition points at which the best-fitplane is changing from waxing to waning.

Notably, detecting the transition point at which the best-fit planechanges from waxing to waning may not be sufficient to conclude that thetransition point corresponds to a rotational exhaustion situation. Forexample, as shown in FIGS. 9( i), 9(ii), 9(iii), 9(iv) and 9(v), a localminimum in w may be an indication that the best-fit plane is changingfrom waxing to waning, but may not indicate the existence of arotational exhaustion situation.

Specifically, as shown in FIG. 9( i), a user has placed three touchpoints 912, 914 and 916 corresponding to three contact points B, A andC, respectively, on the 2D display interface. The user then begins tomove the touch point 912 in a small orbit, moving first to the right anddownwards, as shown in FIG. 9( ii), and then starts to return the touchpoint 912 back towards the touch points 914 and 916, as shown in FIGS.9( iii) and 9(iv). The user returns the touch point 912 to a locationclose to its starting location to complete the small orbit, as shown inFIG. 9( v). The theoretical spheres 924, 926, 928, 930 and 932 track themovement of the normal to be best-fit plane in a similar manner asdescribed previously with respect to FIGS. 8( i)-8(v). As shown in FIG.9( iii), the point at which the touch point 912 begins its return backtowards the touch points 914 and 916 corresponds to a local minimum inw. That is, as shown by sphere 928 of FIG. 9( iii) as compared to sphere926 of FIG. 9( ii) and sphere 930 of FIG. 9( iv), the normal to thebest-fit plane of the contact points A, B and C has become at this pointas parallel to the camera eye vector (i.e., the unit length vector alongthe camera's Z-axis that is located at the closest point of the sphereto the reader) as it will ever be in this interaction. Notably, however,while this interaction does result in a local minimum in w, it does notcorrespond to a rotational exhaustion situation in that no rotationabout an axis has been exhausted that would lead to an unexpected andabrupt change in object orientation.

Given that a detection of a local minimum in w may not correspond to adetection of a rotational exhaustion situation, the 2D display interfacedevice may have to capture additional information about the movement ofthe touch points to distinguish rotational exhaustion situations fromother situations that result in a local minimum in w. That is,rotational exhaustion situations may be distinguished from these othersituations in that rotational exhaustion situations occur when the touchpoint or touch points are moved along a relatively consistent trajectorywhile, in contrast, the other situations occur when the touch point ortouch points are moved along a trajectory that is generallyinconsistent. For example, in the interaction shown in FIGS. 8( i)-8(v),the user moved the touch point 812 along an approximately straight linepath across the 2D display interface from its initial location to thepoint at which w reached a local minimum and rotational exhaustionoccurred. In contrast, in the interaction shown in FIGS. 9( i)-9(v), theuser moved the touch point 912 from its initial location to the point atwhich w reached a local minimum along a path that significantly deviatedfrom a straight line and no rotational exhaustion occurred. Therefore,when detecting rotational exhaustion situations, the 2D display devicemay not only detect when the function w reaches a local minimum but alsomay detect whether the movement of the touch point(s) that triggered thelocal minimum was along a path that does not deviate significantly froma straight line. For example, in some applications, a deviation that isequal to or greater than ten or fifteen degrees from a straight linepath of a touch point may be considered sufficient to conclude that thelocal minimum in w is not indicative of rotational exhaustion.

In detecting rotational exhaustion situations, the 2D display device mayfurther require that the change in w from the touch point's initialposition to the touch point's current position be greater than apredetermined minimum threshold. In some applications, the predeterminedminimum threshold may be, for example, 0.25. This requirement that thechange in w surpasses a predetermined minimum threshold helps protectagainst false positives. Specifically, by requiring that the change in wsurpasses a predetermined minimum threshold, a rotational exhaustionsituation will be detected only if the movement of the touch pointscauses the object to rotate by more than a minimum amount. Depending onthe applications, this additional protection against false positives maybe important because if a minima in w is detected but the total changeof w is small, then this likely indicates that the interaction startednear a minima and thus may not qualify as a true rotational exhaustion.Stated differently, an interaction that starts near minima in w may notqualify as a true rotational exhaustion because a true rotationalexhaustion results i a behavior of the object that is both abrupt andsurprising to the user in response to their interaction with the 2Ddisplay interface. When a user starts rotating an object near theminima, however, the user most likely intends to rotate the objectthrough the minima, and, thus, the behavior of the object as it rotatesthrough the minima is less abrupt and likely expected by the user and,therefore, not surprising to the user. In contrast, when the user startsrotating the object far away from the minima in w, which has a rapidshift in rotational behavior near the minima, the shift in behavior whenthe user moves the object through the minima is likely not intended bythe user and, therefore, may appear abrupt and surprising to the user.Accordingly, whether a situation qualifies as rotational exhaustion maydepend on an inference of the intent of the user by the 2D displayinterface device. In this particular case, the 2D display interfacedevice makes this inference based on the amount of change in w.

FIGS. 10A and 10B are a flow diagram of an example process 1000 formoving simulated 3D objects in response to one or more inputs on asubstantially 2D display interface that “corrects” for rotationalexhaustion (i.e., that suppresses the surprising and unexpected behaviorthat is caused by rotational exhaustion and that may confuse users). Theprocess 1000 may be implemented by a 2D display interface device suchas, for example, a multi-touch display device.

Operations 1002, 1004, 1006, 1008 and 1010 of process 1000 may beperformed, for example, in the same manner described previously withrespect to operations 301, 330, 340, 320 and 350, respectively, ofprocess 300. After having identified the new locations of the touchpoints on the 2D display interface, the 2D display interface devicedetermines a transformation (1012) of the 3D object that minimizes theslippage or error between the screen-space locations of the touch pointsand the screen-space locations of their corresponding contact points.Notably, because rotational exhaustion is an artifact of 3D objectmanipulation and not 2D object manipulations, the transformationdetermined by the solver of the 2D display interface device in operation1012 assumes that the object can move in 3 dimensions (i.e., 6 DOFs) Inone implementation, the 2D display interface device includes a solverthat uses a least-squares minimization algorithm, such as, for example,the LM algorithm, to find a transformation vector q that minimizes theenergy equation E in the same manner as described previously withrespect to operation 360 of process 300. The transformation vector qfound by the 2D display interface device may then be used to transformthe 3D object (1014). The 2D display interface device determines whetherthe interaction (i.e., the placing of the touch points and thesubsequent movement of one or more of the placed touch points)corresponds to a rotational exhaustion situation (1016).

FIG. 11 is a flow diagram illustrating an example implementation ofoperation 1016 of process 1000 that may be performed by the 2D displayinterface device to detect rotational exhaustion situations. The 2Ddisplay interface device finds a best-fit plane for the contact pointsof the 3D object after the 3D object has been transformed (e.g.,transformed using the transformation vector q) (1110).

The 2D display interface device determines whether the transformation ofthe 3D object results in the best-fit plane transitioning from waxing towaning (1120). In one implementation, the 2D display interface devicemakes this determination by determining whether the function w hasreached a local minimum in time. If the function w has not reached alocal minimum in time, then the best-fit plane is not transitioning fromwaxing to waning, and, consequently, rotational exhaustion is notdetected. The 2D display interface device, therefore, renders thetransformed 3D object of the 2D display interface for viewing by theuser (1018). If, however, the function w has reached a local minimum intime, then the best-fit plane is transitioning from waxing to waning,and, consequently, rotational exhaustion may be occurring.

Given the possibility that rotational exhaustion may be occurring, the2D display interface device identifies the touch point or touch pointsin motion that caused the best-fit plane to transition from waxing towaning (1130). The 2D display interface device determines whether themovement of the identified touch point or touch points from theirinitial screen-space locations to their current screen-space locationsresulted in a change in the angle between the normal of the best-fitplane and the camera eye vector that surpasses a minimum threshold(1140). In one implementation, this determination may be accomplished bycalculating the difference between a first value for w that correspondsto the orientation of the object when the touch points were at theirinitial screen-space locations and a second value for w that correspondsto the orientation of the object when the touch points are located attheir current screen-space locations. The initial screen-space locationof a touch point may be, for example, the screen-space location of thetouch point when the touch point was first placed on the 2D displayinterface by the user. Additionally or alternatively, the initialscreen-space location of a touch point may be the screen-space locationof the touch point at a predetermined time prior to the movement of thetouch point that led to the potential rotational exhaustion situation(e.g., 5 seconds prior).

If the change in the angle does not surpass the minimum threshold, thenrotational exhaustion is not detected, and the 2D display interfacedevice, therefore, renders the transformed 3D object on the 2D displayinterface for viewing by the user (1018). If, however, the change in theangle surpasses the minimum threshold, then rotational exhaustion may beoccurring, and, therefore, 2D display interface device may determinewhether the trajectory or path taken by each identified touch point fromtheir initial screen-space location to their current screen-spacelocation deviates significantly from a straight line (1150).

If the trajectory or path of at least one of the identified touch pointsdeviates significantly from a straight line, then rotational exhaustionis not detected, and the 2D display interface device, therefore, rendersthe transformed 3D object on the 2D display interface for viewing by theuser (1018). If, however, the trajectory of each of the identified touchpoints does not deviate significantly from a straight line and,therefore, represents a consistent and smooth movement, then the 2Ddisplay interface device concludes that rotational exhaustion isoccurring and, therefore, may determine a new transformation of theobject that corrects the undesirable object movement caused by therotational exhaustion (1020).

In some implementations of process 1100, the order of operations1120-1150 may be changed. For example, operation 1150 may be performedby the 2D display interface device before operation 1160.

Referring back to FIGS. 10A and 10B, if rotational exhaustion isdetected, the 2D display interface device may determine a correctedtransformation of the 3D object that corrects the rotational exhaustionwhile minimizing the error between the screen-space locations of thetouch points and the screen-space locations of the contact points(1020). In one implementation, the 2D display interface devicecalculates a corrected transformation vector q by using the same solverthat was used calculate the transformation vector q for operation 1012.However, when determining the corrected transformation vector q, thesolver now limits the DOFs that the minimization algorithm can operateupon. That is, rather than allowing the minimization algorithm tooperate upon all 6 DOFs for finding the transformation vector q, as wasdone in operation 1012, the solver will now force the minimizationalgorithm to operate on less than 6 DOFs. This limitation of DOFs resultin the solver calculating a corrected transformation vector q thatprevents the object from translating or rotating in the abrupt andunexpected manner that arises from rotational exhaustion.

Given that rotational exhaustion occurs from exhausting rotational DOF,it may seem natural to constrain all rotational DOFs of the object(i.e., constrain object rotation about the x, y and z axes) to theircurrent values in order to correct the rotational exhaustion problem.Such a constraint results in the rotation of the object stopping beforeunexpected behavior occurs. For example, the solver, when applying theleast-squares minimization algorithm to calculate the correctedtransformation vector q, may fix qx, qy and qz to their currenttransformation values (i.e., to the values corresponding to theorientation of the object immediately prior to the touch point movementthat triggered the rotational exhaustion) such that they may not bevaried by the solver when calculating the vector q that results in theminimum error. However, in practice, such a constraint also causes themovement of thee object being controlled to be limited to translationand thus the interaction may feel “stiff” to the user, as if the objectis sliding on a track.

As described previously with respect to FIGS. 8( i)-8(v), the unexpectedand sudden movement of the object when an axis of rotation has beenexhausted involves not only a change in the axis of rotation but alsoinvolves a translation along the Z-axis of the camera. This translationworks hand-in-hand with the new rotation in order for the touch pointsand the contact points to be matched. Therefore, another way to correctthe rotational exhaustion problem is to restrict translation along theZ-axis of the camera to its current transformation value. Such aconstraint also results in the rotation of the object stopping beforeunexpected behavior occurs. For example, the solver, when applying theleast-squares minimization algorithm to calculate the correctedtransformation vector q, may fix tz to its current transformation value(i.e., to the value corresponding to the depth of the object along theZ-axis immediately prior to the touch point movement that triggered therotational exhaustion) such that tz may not be varied by the solver whencalculating the vector q that results in the minimum error. In practice,constraining movement of the object along the Z-axis may provide a morenatural feel to the user than restricting the rotational DOFs. Forexample, the object may appear to the user as being suspended in spacerather than stuck on a track.

After the corrected transformation is found, the 2D display interfacedevice transforms the object using the corrected transformation (e.g.,the corrected transformation vector q) (1022) and renders thetransformed object (1024). The 2D display interface device then performsoperations 1026, 1028, 1030, 1032 and 1034 to continue monitoring forand responding to the placement and movement of touch points in a mannerthat may be identical to that described previously with respect tooperations 310, 330, 340, 320 and 350, respectively, of process 300.

When one or more touch points have been moved by the user such thatanother transformation of the object is triggered, the 2D displayinterface device may identify the new locations of the touch points onthe 2D display interface (1034) and may continue to determine acorrected transformation of the object (i.e., a transformation of theobject that uses less than all 6 DOFs) to suppress rotational exhaustion(1036). However, eventually the user may move the touch points such thatrotational exhaustion is no longer a problem. The 2D display interfacedevice is able to detect when this occurs by determining whether theminimized error/energy E that results from the corrected transformationhas fallen beneath a predetermined threshold (1038). The predeterminedthreshold may vary by application and depends on a decision made by theapplication designer regarding when a deviation between the contactpoints and the touch points has been reduced to the extent that it isdeemed “close enough” to the nonslippage (i.e., 6 DOF) positions thatrotational exhaustion is no longer considered a problem. Since energymeasures a deviation in pixels, the application designer may, forexample, specify or input into the 2D display interface device a numberof pixels in deviation per contact point that is considered close enoughto the nonslippage contact point positions that rotational exhaustion isno longer considered a problem for the interaction. The 2D displayinterface device then may calculate the predetermined threshold from thespecified deviation. For example, if a deviation of five pixels percontact point is considered close enough to the nonslippage position andthree touch points are currently placed on the 2D display interface, the2D display interface device may calculate the predetermined threshold tobe 3×5²=75 units of energy.

Notably, the minimized error/energy must fall beneath the predeterminedthreshold, not just be beneath the predetermined threshold, for all 6DOFs to be re-enabled. This distinction is significant because thecondition to re-enable all 6 DOFs is not simply that the energy is belowa predetermined threshold but rather that the energy is below thepredetermined threshold AND decreasing. The decreasing aspect isimportant because, when initially disabling DOFs for correcting therotational exhaustion situation, the energy will become very low (i.e.,close to zero) and may, in fact, be below the predetermined threshold.However, as the interaction continues, the energy will increase ratherthan decrease (i.e., the interaction will be incurring more and moreerror or slippage), and, therefore, the energy will not satisfy thecondition that it fall beneath the predetermined threshold. Only afterthe energy is increased above the predetermined threshold, and the userthen brings the object “close enough” to its original pre-slippageconfiguration does the energy decrease below (i.e., fall beneath) thepredetermined threshold and trigger all 6 DOFs of the object to bere-enabled.

If the minimized error/energy E exceeds the predetermined threshold,then rotational exhaustion is deemed by the 2D display interface deviceto still be a problem and the object is transformed using the correctedtransformation (1022) and rendered (1024). If the minimized error/energyE falls below the predetermined threshold, the 2D display interfacedevice concludes that rotational exhaustion is no longer a problem and anew transformation of the object is determined using all 6 DOFs (1012).The object is then transformed using the new transformation (1014) andrendered (1018). In another implementation, the 2D display interfacedevice may detect that the rotational exhaustion is no longer a problemby detecting when the touch points are moved back close to theirpre-slippage positions by the user rather than by comparing theenergy/error to a threshold value.

The process 1000 is similar to the process 300 in that it triggerstransformation of the 3D object in response to movement of touch points.Other implementations of the 2D display interface device mayadditionally trigger the transformation of the 3D object in response toadding or removing touch points or in response to applying a differentpressure to one or more existing touch points. Alternatively, the 2Ddisplay interface device may continually (or periodically) transform the3D object at fixed time increments (e.g., every 1 millisecond),irrespective of movement, addition, or removal of touch points or ofpressure changes at touch points.

As described previously with respect to FIGS. 5A-5E, the solver of the2D display interface device may not always be able to determine atransformation of the 3D object that results in no error of slippage(e.g., E=0). For example, when using the LM algorithm, the number ofterms in the energy function (i.e., the number of input DOFs) may haveto be equal to or greater than the number of DOFs of the object. If thenumber of terms in the energy function is greater than the number ofDOFs of the object, then slippage or error occurs. That is, the minimumerror/energy transformation vector q found by the solver will correspondto E>0.

As described previously with respect to operation 360 of process 300,each touch point introduces two terms into the energy function (i.e.,energy or error in x and energy or error in y). Therefore, for anyinteraction that involves four or more touch points, the energy functionwill be overconstrained and slippage will occur. That is, the energyfunction will have eight or more terms, which is greater than themaximum number of DOFs (i.e., six) available to a 3D object movingfreely in world-space. Similarly, if the 2D display interface devicereduces the number of DOFs of the object, then the energy function E maybe overconstrained and slippage may occur even when responding to threetouch point interactions. For example, if translation of the objectalong the Z axis is prevented, then the object will only have five DOFs,which is less than the six terms introduced into the energy function bythe three touch points.

The energy function introduced previously with respect to operation 360of process 300 is shown again below:

$E = {\sum\limits_{i}^{\mspace{11mu}}\; {{{s\left( {x_{i},q} \right)} - p_{i}}}^{2}}$

Since error is measured by this energy function in L² (i.e., in thesquare of the Euclidean distances), the error tends to be distributedrelatively evenly among the touch points. In other words, the Euclideandistance from the screen-space location of a contact point to thescreen-space location of its matching touch point is about the same foreach contact point/touch point pair.

FIGS. 12A, 12B and 12C are diagrams illustrating displays presented to auser by a 2D display interface that show slippage between touch pointsand their corresponding contact points. Referring to FIG 12A, a userpreviously placed touch points 1210, 1212 and 1214 onto a displayed cube1200, and the touch points were previously matched by the 2D displayinterface device with contact points 1216, 1218 and 1220 on the cube1200. The user then kept the location of the touch points 1210 and 1212fixed while moving the touch point 1214 in a rightward direction awayfront the fixed touch points 1210 and 1212 to the position shown in FIG12A. In this example, the application executed by the 2D displayinterface device prevents the cubed 1200 from translating along theZ-axis of the camera, and, therefore, the movement of the touch point1214 has resulted in slippage between the touch points and theirrespective contact points. As shown in FIG. 12A, the slippage or errorhas been distributed evenly among the touch points such that no contactpoint is displayed on the 2D display interface at the same location asits corresponding or matched touch point. FIG. 12A may, for example,correspond to an error distribution among the touch points that wouldarise when minimizing the energy function introduced previously withrespect to operation 360 of process 300.

In some applications, however, a uniform distribution of the error amongthe touch points may not be desirable. By adding a normalized weightingcoefficient to each contact points terms in the energy function, thedistribution of the error among the touch points can be controlled. Thisadditional coefficient may be indicative of how “important” a particulartouch point is relative to the other touch points and, thus, mayindicate the degree to which that touch point will be matched during theoptimization process (i.e., during the energy or error minimizationprocess performed by the solver of the 2D display interface device). Ifω_(i) is the weight of the ith touch point, then the new energy equationwith weighting coefficients included becomes:

$\begin{matrix}{E = {\sum\limits_{i}^{\mspace{11mu}}\; {\omega_{i}{{{s\left( {x_{i},q} \right)} - p_{i}}}^{2}}}} \\{\sum\limits_{i}\; \omega_{i = 1}}\end{matrix}$

The 2D display interface device may use various different schemes todefine the weights of the touch points. The choice of the schemetypically depends on the application that is being executed by the 2Ddisplay interface device. In some implementations, the 2D displayinterface device uses a scheme that dynamically assigns weights to touchpoints while the user in interacting with the 2D display interface basedon some measurable property of the interaction.

For example, in some implementations, the 2D display interface devicemay assign the appropriate weighting coefficients to touch points basedon the screen-space distance a touch point has travelled from itsinitial position to its current position on the 2D display interface.Screen-space distances may be well-suited for being used to assign touchpoint weighting coefficients because a touch point's motion may beviewed as an indication of the user's intent. If a touch point is movedquite a bit by the user while the other touch points remain relativelyfixed, then the touch point in motion may be considered more importantthan the others, and thus should be matched to its corresponding contactpoint more closely. If weighting is defined in this manner, then whentranslation along camera Z is disabled, the object may translate tofollow the dominant hand's fingers (i.e., touch points) across thescreen while the non-dominant hand's fingers (i.e., touch points) exertlittle influence on the object's motion.

FIG. 12B illustrates a display of touch points that have been weightedusing this weighting scheme. The touch points 12010, 1212 and 1214 shownin FIG. 12B have been moved exactly as described previously with respectto FIG. 12A but now the locations of their corresponding contact points1216, 1218 and 1220 have changed. Specifically, because the touch point1214 moved while the touch points 1210 and 1212 remained fixed, thescreen-space location of contact point 1220 is now almost identical tothat of its matched touch point 1214 (i.e., almost none of the error isdistributed to touch point 1214) while the screen-space locations ofcontact points 1216 and 1218 are now even further away from theirmatched touch points 1210 and 1212 (i.e., almost the whole error isdistributed among touch points 1210 and 1212). Using such a weightingscheme, for example, the 2D display interface device may assign a weightof w₁ to a touch point that is equal to some value that is directlyproportional to the distance traveled by the touch point. In thescenario shown in FIG. 12B, the 2D display interface device may, forexample, have assigned a weight w₁ equal to 0.98 to the two energy termscorresponding to touch point 1214, a weight _(w2) equal to 0.01 to thetwo energy terms corresponding to touch point 1210, and a weight w₃equal to 0.01 to the two energy terms corresponding to touch point 1212.

In other implementations, the 2D display interface device may defineweights such that stationary touch points are more important than touchpoints in motion. As a result, for an object unable to translate alongthe Z-axis, the stationary touch points pin the object down while thetouch points in motion exert only a small rotational influence on thepinned object. Thus, this weighting scheme maintains the object'sposition and orientation at the time the slippage occurred.

FIG. 12C illustrates a display of touch points that have been weightedusing this weighting scheme. The touch points 1210, 1212 and 1214 shownin FIG. 12C have been moved exactly as described previously with respectto FIG. 12A but now the locations of their corresponding contact points1216, 1218 and 1220 have changed. Specifically, because the touch points1210 and 1212 remained stationary while the touch point 1214 moved, thescreen-space location of contact points 1216 and 1218 are now almostidentical to that of their matched touch points 1210 and 1212,respectively (i.e., almost none of the error is distributed to touchpoints 1210 and 1212) while the screen-space location of contact point1220 is now even further away from its matched touch point 1214 (i.e.,almost the whole error is distributed to touch point 1214). Using such aweighting scheme, for example, the 2D display interface device mayassign a weight of w₁ to a touch point that is equal to some value thatis inversely proportional to the distance traveled by the touch point.Additionally or alternatively, the 2D display interface device mayassign to every touch point that is stationary a first predeterminedfixed weight and to every touch point that is in motion a secondpredetermined fixed weight that is much smaller than the firstpredetermined fixed weight. Notably, the first and second predeterminedfixed weights are typically normalized by the 2D display interfacedevice prior to being used in the energy equation E. In the scenarioshown in FIG. 12C, the 2D display interface device may, for example,have assigned a weight w₁ equal to 0.02 to the two energy termscorresponding to touch points 1214, a weight w₂ equal to 0.49 to the twoenergy terms corresponding to touch point 1210, and a weight w₃ equal to0.49 to the two energy terms corresponding to touch point 1212.

In some implementations, when applying the above-described weightingtechniques, the 2D display interface device may calculate the distance atouch point has traveled from an initial location to its currentlocation by defining the initial location to be the location of thetouch point when it was first placed on the 2D display interface by theuser. Additionally or alternatively, the 2D display interface device maycalculate the distance by defining the initial location to be thelocation of the touch point a predetermined time prior to the time atwhich the distance calculation is performed.

In some implementations, the 2D display interface device may calculatethe distance a touch point has traveled from an initial location to itscurrent location as the 2D Euclidean straight line distance between thetwo locations. In other implementations, the 2D display interface devicemay calculate the distance a touch point has traveled from an initiallocation to its current location as an arc length that represent thetotal distance the touch point actually traveled between the twolocations.

When weighting using are lengths, important points may be “remembered”because the 2D display interface device may maintain a record of allmotion of touch points throughout the interaction. As such, once a touchpoint becomes important, it may remain important. Consequently, a movingtouch point may continue to be weighted highly by the 2D displayinterface device even when it returns close to its original locationbecause the total distance it traveled on its path during theinteraction was very long relative to that of the other touch points.Notably, even when the touch points is moved back to the location whereslippage was originally triggered, the touch point may be the only oneto have any effect on the object being controlled due to the heavyweight assigned to it by the long path distance that it traveled.Additionally, if one touch point moves much more than the other touchpoints (such that its arc length or path length is very long) and thenslows to a rest while the other touch points start to move, it may takesome time before the newly moving touch points are able to exert anyinfluence at all on the object's orientation.

In some implementations,. pressure also may be used to control thespread of error among the touch points. By defining a touch point'sweight in terms of how hard a user is pressing against the screen, the2D display interface device may provide the user with a tool thatenables the user to pin touch points down. The harder a user presses ata touch point relative to how he/she presses at other touch points, theless error will be allocated to that touch point by the solver. In otherwords, the harder a user presses at a touch point relative to how he/shepresses at other touch points, the closer the screen-space location ofthe matching contact point will be to the screen-space location of thetouch point after the object is transformed by the 2D display interfacedevice. Because pressure values may be continuous and adjustable, touchpoints may be re-weighted dynamically on the fly. Using such a weightingscheme, for example, the 2D display interface device may assign a weightof w_(i) equal to some value that is directly proportional to therelative pressure applied at the touch point. The relative pressure fora given touch point may be, fore example, a percentage value thatrepresents the currently applied pressure at the touch point as apercentage of the average pressure currently being applied by the userat all of the touch points or, alternatively, as a percentage of theminimum pressure of all the pressures currently being applied by theuser at the touch points.

Using pressure values at touch points to control error distribution,however, may become problematic for 2D display interface devices thatrequire the user to use his/her fingers to place and move touch pointson the 2D display interface by physically pressing a surface of the 2Ddisplay interface. Specifically, when a user presses his/her finger hardagainst a touch-screen and moves the finger at the same time, frictionmay cause the interaction to be unpleasant to the user and thusundesirable. Therefore, in some implementations, pressure values areused to control error distribution by making touch points sticky ratherthan by more heavily weighting touch points that are in motion. In theseimplementations, for example, the 2D display interface device may assigna first predetermined fixed weight (or a weight that varies directlywith applied relative pressure) to every touch point that is bothstationary and at which the relative pressure applied by the user isgreater than a predetermined threshold, and may assign to every othertouch point a second predetermined fixed weight that is much smallerthan the first predetermined weight (or much smaller than the minimumweight assigned to a stationary touch point at which a relative appliedpressure exceeds the predetermined threshold). Notably, the first andsecond predetermined fixed weights are typically normalized by the 2Ddisplay interface device prior to being used in the energy equation E.

In some implementations, multiple weighting schemes are concurrentlyused. For example, the 2D display interface device may assign theweighting coefficients to touch points based on the screen-spacedistance a touch point has travelled from its initial position to itscurrent position on the 2D display interface and, for fixed touchpoints, based on the relative pressure applied.

FIG. 13 is a flow diagram of an example process 1300 for movingsimulated 3D objects in response to one or more inputs on asubstantially 2D display interface that distributes slippage errorsamong touch points in a controlled manner. The process 1300 may beimplemented by a 2D display interface device such as, for example, amulti-touch display device.

Operations 1310, 1320, 1330, 1340 and 1350 of process 1300 may beperformed, for example, in the same manner described previously withrespect to operations 310, 320, 330, 340 and 350, respectively, ofprocess 300. After having identified the new locations of the touchpoints on the 2D display interface, the 2D display interface devicedetermines weighting values for the touch points (1352). The 2D displayinterface device may determine the weighting values based oncharacteristics of the touch points such as, for example, the distancetraveled by the touch point from an initial location, whether the touchpoint is stationary or moving, and/or the pressure applied by the userat the touch point. The 2D display interface device may, for example,use any of the weighting schemes described previously, either alone orin combination with other weighting schemes, to determine theappropriate weighting value for each touch point.

The 2D display interface device normalizes the weighting values todetermine weighting coefficients (1354) and then determines atransformation of the 3D object that minimizes the slippage or errorbetween the screen-space locations of the touch points and thescreen-space locations of their corresponding contact points using theweighting coefficients (1360). Operation 1360 may be performed, forexample, in the same manner as described previously with respect tooperation 360 of process 300 with the exception that the terms of theenergy function that is being minimized now include the weightingcoefficients to distribute the error or slippage among the differenttouch points. In some implementations, the 2D display interface deviceapplies an algorithm to minimize the previously described energyfunction E with weighting coefficients. The 2D display interface devicethen transforms the 3D object using the determined transformation (1370)and renders the transformed 3D object for display on the 2D displayinterface (1380).

Depending on the application of the 2D display interface device, it maybe desirable to constrain an object's motion to meet task-specific ends.Constraints on an object's motion have been described previously withrespect to operation 360 of process 300 and with respect to solvingambiguous rotation and rotational exhaustion situations. Up to now, thedescribed constraints on the movement of the 3D object have been limitedto reducing the DOFs of the object (e.g., in order to provide exactsolutions for one and two point interactions) and adding an additionalpenalty term to the energy equation that further constrains the 3Dobject's movement by requiring that the 3D object, after transformation,have a depth value for one or more contact points set by the pressureapplied by the user at the corresponding touch points. This latterconstraint was referred to as a penalty and is distinct from the formerconstraint in that the DOFs of the object upon which the solver operatesto determine a minimum energy/error transformation vector q are notreduced by the penalty (i.e., the object still has 6 DOFs, assuming noother application-specific DOF reductions have been applied).

A penalty or a penalty function is a function that measures deviationfrom some ideal condition. The penalty function previously introducedwith respect to solving ambiguous rotations, for example, measureddeviation of the z coordinate of a contact point from a z_(target) setby the user through application of a pressure at the touch pointcorresponding to the contact point. In another example, a penaltyfunction may be used in an application that allows a globe to be spun bya user by enforcing the condition that the globe's center always staysat the same location in world-space. In such an application, the penaltyfunction may measure the distance of the globe from the globe's originalcenter and may penalize (i.e., highly increase the energy of the energyfunction) any object transformations that the solver attempts to apply,when testing different object transformations for minimizing energy,that correspond to the globe moving from the globe's original center.

A 2D display interface device that uses penalties to constrain themovement of a 3D object and that allows control of the distribution ofslippage among the different touch points may, for example, use thefollowing energy equation:

$E = {{\sum\limits_{i}^{\mspace{11mu}}\; {\omega_{i}{{{s\left( {x_{i} \cdot q} \right)} - p_{i}}}^{2}}} + {\sum\limits_{i}\; {\lambda_{i}{g_{i}(q)}^{2}}}}$

where g_(i)(q) is the j^(th) penalty function with a weight coefficientλ. Penalty functions may not be true, hard constraints of the object'smotion in the strictest sense (unlike, for example, reductions in objectDOFs), but, through a careful choice of penalty terms, it may bepossible to achieve substantially the same results.

As stated previously, the solver of the 2D display interface device maynot always be able to determine a transformation of the 3D object thatresults in no error or slippage (e.g., E=0) when the number of terms inthe energy function (i.e., the number of input DOFs) is greater than thenumber of DOFs of the object. Adding penalty terms to the energyfunction, therefore, may result in sacrificing the exactness of thesolution found by the solver. For example, for three touch pointinteractions, the addition of even one penalty term to the energyfunction can result in slippage (i.e., the energy function will haveseven terms while the DOFs of the object is six). However, for one touchpoint interaction, up to four penalty terms maybe added to the energyfunction without resulting in any error or slippage. Notably, if lessthan four penalty terms are added, then the number of DOFs of the objectmay have to be reduced to a number equal to the sum of two plus thenumber of added penalty terms in order to allow the solver to arrive atan exact solution with no error or slippage (i.e., E=0). For two touchpoint interactions, up to two penalty terms may be added to the energyfunction without resulting in any error or slippage. However, if onlyone penalty term is added, then the number of DOFs of the object mayhave to be reduced to five in order to allow the solver to arrive at anexact solution.

Accordingly, the addition of five or more penalty terms for one touchpoint interactions, three or more penalty terms for two touch pointinteractions, and one or more penalty terms for three touch pointinteractions may result in slippage between the screen-space location ofat least one of the touch points and the screen-space projection of itscorresponding contact point. While a limit may not be imposed on thenumber of penalty terms added to the energy function for a givenapplication, the presence of a large number of penalty terms may causethe interaction to noticeably degrade. That is, the presence of a largenumber of penalty terms may cause significant slippage or error suchthat the desired illusion that the user is actually “gripping” the 3Dobject is significantly compromised.

When energy is overconstrained (i.e., when the number of terms in ourenergy function exceeds the number of available DOF), properly weightingthe penalty functions may be required to ensure that each penaltycontinues to function sufficiently as a constraint. Choosing a value forλ to weight a screen-space penalty may serve to inform the solver of thetrade-off that the application designer finds acceptable betweenviolating the penalty and the amount of contact point slippage that willbe allowed. For example, if screen-space locations are measured bynumber of pixels, a violation or slippage of one pixel against the idealcondition of the penalty is equivalent to λ pixels of slippage of anunweighted contact point. In other words, violating the ideal conditionof the penalty by one pixel corresponds to introducing the same amountof error as deviating the screen-space location of an unweighted contactpoint by λ pixels away from the screen-space location of its matchingtouch point. Therefore, the larger that the application designer makesthe value of λ, the harder the solver of the 2D display interface devicewill work to ensure that the penalty only receives an equivalently smallamount of the error. In implementations that use the energy functiondescribed previously, which corresponds to a quadratic function incontact point slippage, the application designer may, for example, setthe value of λ to be in the range of 10 to 10 ⁴ to keep penaltyviolations small and slippage to a minimum.

The application designer may find that choosing a value of λ to weight aworld-space penalty is more difficult and challenging than choosing avalue for λ to weight a screen-space penalty. This difficulty arisesfrom the fact that the energy function is expressed in terms ofscreen-space, and, therefore, deviations from an ideal condition inworld-space may first have to be projected into screen-space in order todetermine the extent of their corresponding screen-space deviation orerror. Therefore, due to this difference in dimensionality, smallviolations of penalties in world-space are potentially much moredisastrous than small violations of penalties in screen-space. Forexample, a violation of one pixel in screen-space may be barelynoticeable, but a violation of the same magnitude in world-space may beglaring. If an application can only tolerate a violation of 10^(−k) inworld-space of a given penalty before violations become noticeable tothe user, the application designer should set a value for λ for thispenalty to be at least 10^(2k). By setting the value for λ to be10^(2k), violating the ideal condition of the world-space penalty suchthat the violation just becomes noticeable to the user introduces thesame amount of error as deviating the screen-space location of anunweighted contact point by one pixel away from the screen-spacelocation of its matching touch point. The application designer, however,may want the penalty to be more strictly enforced such that violation ofthe penalty carries more significance than slippage among the contactpoints. Therefore, the application designer may instead choose to setthe value for λ for the world-space penalty to be even higher than10^(2k), such as, for example, 10^(2k−1) to 10^(2k+4). To illustrate, ifthe application designer sets the value for λ to be 10^(2k+4), violatingthe ideal condition of the world-space penalty such that the violationjust becomes noticeable to the user introduces the same amount of erroras deviating the screen-space location of an unweighted contact point by10,000 pixels away from the screen-space location of its matching touchpoint.

In setting the value for λ to weight penalties, the application designermay have to be careful to not choose a value for λ that is too large(e.g., greater than 10⁶ to 10⁸). If a penalty is overweighted, thesolver of the 2D display interface device may not be able to converge toa solution because, for example, as it chooses different paths to takein the solution space, it may find that any movement at all may resultin penalty values that dwarf the error associated with contact pointslippage. As a result, the solver may be unable to proceed to asolution, and, as a consequence, the 3D object displayed on the 2Ddisplay interface may stop responding.

An application designer may use the 2D display interface device toconcurrently or separately impose a variety of different types ofpenalties on the motion of an object. For example, the applicationdesigner may define a point “a” in object-space and require that thelocation of the screen-space projection of the point “a” be fixed. Insome implementations, the application designer may incorporate thispenalty into the application executed by the 2D display interface deviceby adding the following penalty term to the energy function:

g(q)=∥b−s(a,q)∥

wherein b is the fixed screen-space projection of the object-space point“a,” which may be calculated, for example, at the beginning of theinteraction, and s(a,q) is the screen-space projection of object-spacepoint “a” after the object-space has been transformed in accordance withthe transformation vector q during the interaction (e.g., by movement ofthe object resulting from, for example, movement of touch points by theuser). Accordingly, in this example, the penalty function is equal tothe Euclidean distance between the fixed screen-space projection of theobject-space point “a” (i.e., b) and the screen-space projection of theobject-space point “a” after the object-space has been transformed inaccordance with the transformation vector q.

In another example, the application designer may define a point “a” inobject-space and require that the world-space location of point “a” befixed. In some implementations, the application designer may incorporatethis penalty into the application executed by the 2D display interfacedevice by adding the following penalty term to the energy function:

g(q)=∥b−M(q)a∥

where b is the fixed world-space location of the object-space point “a,”which may be calculated, for example, at the beginning of theinteraction, and M(q)a is the world-space location of object-space point“a” after the object-space has been transformed in accordance with thetransformation vector q during the interaction. Accordingly, in thisexample, the penalty function is equal to the Euclidean distance betweenthe fixed world-space location of the object-space point “a” (i.e., b)and the world-space location of the object-space point “a” after theobject-space has been transformed in accordance with the transformationvector q.

The application designer may additionally or alternatively place aconstraint on object motion that requires that the screen-space distancebetween points “a” and “b” on the object remains constant. In someimplementations, the application designer may incorporate this penaltyinto the application executed by the 2D display interface device byadding the following penalty term to the energy function:

g(q)=(∥s(a,q)−s(b,q)∥−d)

where d is the fixed screen-space distance between points “a” and “a,”which may be calculated, for example, at the beginning of theinteraction, and s(a,q) and s(b,q) are the screen-space projections ofpoints “a” and “b” on the object, respectively, after the object hasbeen transformed in accordance with the transformation vector q duringthe interaction. Accordingly, in this example, the penalty function isequal to the difference between the fixed screen-space distance betweenpoint “a” and “b” (i.e., b) and the screen-space distance (i.e., the 2DEuclidean distance) between the screen-space projection of object point“a” and the screen-space projection of object point “b” after the objecthas been transformed in accordance with the transformation vector q.

The application designer may additionally or alternatively place aconstraint on object motion that requires that the world-space distancebetween points “a” and “b” on the object remain constant. In someimplementations, the application designer may incorporate this penaltyinto the application executed by the 2D display interface device byadding the following penalty term to the energy function:

g(q)=(∥M(q)a−M(q)b∥−d)

where d is the fixed world-space distance between points “a” and “b,”which may be calculated, for example, at the beginning of theinteraction, and M(q)a and M(q)b are the world-space positions of points“a” and “b” on the object, respectively, after the object has beentransformed in accordance with the transformation vector q during theinteraction. Accordingly, in this example, the penalty function is equalto the difference between the fixed world-space distance between points“a” and “b” (i.e., d) and the world-space distance (i.e., the 3DEuclidean distance) between the world-space location of object point “a”and the world-space location of object point “b” after the object hasbeen transformed in accordance with the transformation vector q.

The application designer may additionally or alternatively place aconstraint on object motion that restricts the rotation of the objectabout an axis “v” in world-space. In some implementations, theapplication designer may incorporate this penalty into the applicationexecuted by the 2D display interface device by adding the followingpenalty term to the energy function:

g(q)=

v, M(q)w

where v is the axis in world-space, which may be selected, for example,at the beginning of the interaction, w is the same selected axis inobject-space, M(q) is (as defined previously) the object-space toworld-space transformation matrix, and the < > indicate an inner or dotproduct. Accordingly, in this example, the penalty function is equal tothe dot product of the fixed axis in world-space with the same axis inobject-space after the object has been transformed in accordance withthe transformation vector q during the interaction. As is evident fromthis penalty function, minimization of the penalty requires that thesolver maintain the axis w in object-space as parallel as possible withthe fixed axis in world-space.

The application designer may additionally or alternatively place aconstraint on object motion that requires that the world-space locationof object point “a” be restricted to a plane. In some implementations,the application designer may incorporate this penalty into theapplication executed by the 2D display interface device by adding thefollowing penalty term to the energy function:

g(q)=

n,M(q)a

where n is the normal to the selected plane in world-space, which may bedetermined from a plane selected, for example, at the beginning of theinteraction, M(q) is the object-space to world-space transformationmatrix, a is the location of the point “a” in object-space, and the < >indicate an inner or dot product. Accordingly, in this example, thepenalty function is equal to the dot product of the normal to theselected plane in world-space with the world-space location of the point“a.”

Notably, when adding the penalty terms described above to the energyfunction, the terms are typically squared and multiplied by λ.Accordingly, the contribution of these penalties to the energy or erroris equal to the square of the deviation from their ideal conditionweighted by λ.

As illustrated by the aforementioned examples, many different types ofpenalties can be added to the energy function to achieve various typesof restrictions on the object's movement. In general, an applicationdesigner has a lot of flexibility to design virtually any kind ofrestriction to object motion by leveraging the ability of the disclosedtechniques to transform parameters from object-space to world-space andto screen-space. Given this flexibility, an application designer canimpose restrictions using, for example, distance measurements and dotproducts in any of the three different spaces (i.e., screen-space,world-space and object-space). Once such penalties are implemented, richinteractions may be created with just a few lines of code.

For example, FIG. 14 is a sequence of diagrams that depict a two touchpoint interaction in which a penalty has been added to the energyfunction that only allows rotation of the object 1410 about the firsttouch point 1420 placed by the user on the 2D display interface 1400. Inthis case, the user placed another touch point 1430 after having placedthe first touch point 1420. Notably, in this interaction, the rotationof the object 1410 is about a normal to the plane tangential to thesurface of the object 1410 at the location of the first touch point1420. This type of penalty enables two touch point rotation aboutarbitrary 3D axes since the axis is determined dynamically based on thefirst touch point that is placed on the 2D display interface by theuser.

In some implementations, multiple penalties can be used in tandem toproduce more interesting interactions than a single penalty might allowalone. For example, penalizing a point's deviation from a plane alonemay not stop the rest of an object from orienting itself in somearbitrary manner with respect to the plane. By additionally penalizingthe rotation of the point's normal and tangent vectors, the object maybe forced to slide along the plane without any twisting.

FIG. 15 is a sequence of diagrams that illustrate an example of a singlepoint interaction in which a cube 1510 is constrained to move in a planeperpendicular to the normal of a single touch point 1520 placed on a 2Ddisplay interface 1500, where twisting is not allowed. Notably, theplane perpendicular to the normal is the plane tangential to the surfaceof the cube 1510 at the location of the touch point 1520.

FIG. 16 is a low diagram of an example process 1600 for moving simulated3D objects in response to one or more inputs on a substantially 2Ddisplay interface that restrains movement of the 3D objects usingpenalties and distributes slippage errors among touch points in acontrolled manner. The process 1600 may be implemented by a 2D displayinterface device such as, for example, a multi-touch display device.

Operations 1610, 1620, 1630, 1640 and 1650 of process 1600 may beperformed, for example, in the same manner described previously withrespect to operations 310, 320, 330, 340 and 350, respectively, ofprocess 300. After having identified the new locations of the touchpoints on the 2D display interface, the 2D display interface devicedetermines weighting values for the penalties and for the touch points(1652). The choice of penalties to be applied for a given interaction isdetermined by the application and is typically fixed. However, in someimplementations, the application may dynamically change the choice ofpenalties based on characteristics of touch points or of theinteractions. In such implementations, the 2D display interface devicemay, for example, choose the appropriate penalty functions afteroperations 1650 and prior to or at the same time as operation 1652.

The weighting values for the penalties are also typically fixed topredetermined values by the application in accordance with theprinciples discussed previously. However, in some implementations, theweighting values for the penalties, like those of the touch points, mayvary based on the characteristics of one or more of the touch pointssuch as, for example, the distance traveled by one or more of the touchpoints from an initial location, whether one or more of the touch pointsare stationary or moving, and/or the pressure applied by the user at oneor more of the touch points. The 2D display interface device may, forexample, use any of the touch point weighting schemes describedpreviously, either alone or in combination with other weighting schemes,to determine the appropriate weighting value for each penalty. The 2Ddisplay interface device also may determine the weighting value for thetouch points, for example, in the manner described previously withrespect to FIG. 13.

The 2D display interface device normalizes the touch point and penaltyweighting values to determine weighting coefficients (1654). Anormalized weighting coefficient for a penalty or touch point may be,for example, equal to the weighting value for the penalty or touch pointdivided by the sum the weighting values for all of the penalties andtouch points.

The 2D display interface device then determines a transformation of the3D object that minimizes the slippage or error between the screen-spacelocations of the touch points and the screen-space locations of theircorresponding contact points using the weighting coefficients (1660).Operation 1660 may be performed, for example, in the same manner asdescribed previously with respect to operation 360 of process 300 withthe exception that the terms of the energy function that is beingminimized now include the weighting coefficients to distribute the erroror slippage among the different touch points and penalties. In someimplementations, the 2D display interface device applies an algorithm tominimize the previously described energy function E with weightingcoefficients. The 2D display interface device then transforms the 3Dobject using the determined transformation (1670) and renders thetransformed 3D object for display on the 2D display interface (1680).

In implementations where the object under control is much larger thanthe camera's field of view, the disclosed techniques also may be appliedto navigation tasks. In such implementations, users may leveragepreexisting multi-touch (or, more generally, multi-point) experiences toperform new multi-touch tasks (or, more generally, point task), such as,for example, traversal through virtual environments. FIG. 17 shows auser performing RST-style interactions with a 2D display interface 1700to navigate into a complicated model 1710.

For objects that are not free-floating, the addition of some simplepenalties may be employed by the application designer to ensure desiredinteractions. For example, as described previously, the applicationdesigner may constrain the center of a globe to always remain at thesame location in world-space to thereby allow global navigation by theuser. For single touch point interactions, the application executed bythe 2D display interface device may make the globe rotate underneath theuser's fingertip (or, more generally, touch point) by penalizing changesin the distance between the camera and the center of the globe. Sincethe user can then spin the globe with a single finger (or, moregenerally, touch point) without affecting the camera's distance to theglobe, the presence of these two penalties may allow the disclosedtechniques to mimic an Arcball controller. This interaction stillbehaves correctly at arbitrary camera orientations, such as is shown inFIG. 18, which depicts a single point interaction with a simulated globe1805 displayed in a 2D display interface 1800.

For terrain navigation interactions involving two or more touch points,the distance penalty may be dropped by the application designer. Theuser of the 2D display interface device may be able to zoom in and outwith two fingers, and tilt the globe or change perspective with three ormore fingers. Because the 2D display interface device operates inscreen-space, terrain interactions, such as pinning one point on thesurface down to a particular location on screen, and then spinning theearth around it may be permitted. FIG. 19 is a sequence of diagrams thatdepict this type of interaction. Specifically, as shown in FIG. 19, theuser has pinned down a mountain 1910 displayed on a globe 1905 in the 2Ddisplay interface 1900 and is spinning the globe 1905 around it. Asshown in the sequence of diagrams of FIG. 19, the position of themountain does not change on the 2D display interface 1900.

The described methods, and techniques may be implemented by a 2D displayinterface device embodied in digital electronic circuitry, computerhardware, firmware, software, or in combinations of these elements. The2D display interface device implementing these techniques may be anapparatus that includes appropriate input and output devices, a computerprocessor, and a tangible computer-readable storage medium on which acomputer program or other computer-readable instructions are stored forexecution by one or more processing devices (e.g., a programmableprocessor).

A process embodying these techniques may be performed by a 2D displayinterface device that includes a programmable processor executing aprogram of instructions to perform desired functions by operating oninput data and generating appropriate output. The techniques may beimplemented in one or more computer programs that are executable o a 2Ddisplay interface device that is a programmable system including atleast one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. Each computer program may be implemented in a high-levelprocedural or object-oriented programming language, or in assembly ormachine language if desired; and in any case, the language may be acompiled or interpreted language.

Suitable processors include, by way of example, both general and specialpurpose microprocessors. Generally, a processor will receiveinstructions and data from a read-only memory and/or a random accessmemory. Storage devices suitable for storing computer programinstructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices, such asErasable Programmable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Anyof the foregoing may be supplemented by, or incorporated in,specially-designed application-specific integrated circuits (ASICs).

The various techniques for manipulating displayed content describedherein generally are described in the context of 2D display interfacedevices that are multi-touch display devices, which are configured toreceive multiple inputs at the same time.

The various different techniques for manipulating displayed contentdescribed herein may be implemented both on multi-touch display devicesthat require physical contact with the surface of the multi-touchdisplay device in order to receive input and multi-touch display devicesthat do not require physical contact with the surface of the multi-touchdisplay device in order to receive input. For example, the displayedcontent manipulation techniques described herein may be implemented onmulti-touch display devices that receive input by detecting contact withthe surface of the multi-touch display device by a finger, a stylus,some other mechanical, electro-mechanical, or magnetic input mechanismand/or any combination of multiple such input mechanisms at the sametime. Additionally or alternatively, the various different displayedcontent manipulation techniques described herein may be implemented onmulti-touch display devices that receive input by detecting the presenceof an input mechanism in the vicinity of the surface of the multi-touchdisplay device but that that do not necessarily require physical contactto be made with the surface of the multi-touch display device to receiveinput. Such multi-touch display devices may be configured to receiveinput by detecting the presence of a finger, a stylus, some othermechanical, electro-mechanical, or magnetic input mechanism and/or anycombination of multiple such input mechanisms in the vicinity of thesurface of the multi-touch display device even when such inputmechanisms are not in physical contact with the surface of themulti-touch display device.

The various different techniques for manipulating displayed contentdescribed herein also may be implemented in any other type ofmulti-point computing system configured to receive multiple inputs atthe same, including, for example, systems configured to receiveconcurrent input from multiple pointing devices (e.g., multiple computermice) and/or concurrent input from one or more pointing devices andanother input device (e.g., a keyboard). Another example of amulti-point input computing system within which the controls describedherein may be implemented is a multi-point input capable standalonetablet without an integrated display.

In implementations in which variations in pressure applied at one ormore points on the 2D display interface are used to inform and affectmanipulation of displayed content, the 2D display interface device mayinclude a pressure sensor capable of differentiating between differentlevels of applied pressure. In some implementations, the pressure sensormay detect the size of a contact area (i.e., the surface area of thecontact area) at a touch point, and the 2D display interface device mayestimate an applied pressure value at that touch point based on thedetected contact area size. For example, when fingers are used to placethe touch points, the larger the size of the contact area of a finger ata touch point, the larger the value of applied pressure at that touchpoint. The pressure sensor may, for example use optical,electromagnetic, piezoelectric and/or capacitive sensing technologies todetect the size of the contact area or to otherwise detect anddifferentiate between different levels of applied pressure at a touchpoint.

Various modifications may be made. For example, useful results still maybe achieved if steps of the disclosed techniques are performed in adifferent order and/or if components of the disclosed systems arecombined in a different manner and/or replaced or supplemented by othercomponents. Furthermore, the addition of penalties specified by metricssuch as the speed of the user's fingers may be useful in guiding theoptimization (i.e., error reduction or minimization) process into morefaithfully capturing the user's intent. Additionally, techniques toimplement “undo” operations, by back tracking through solution space canbe performed. Finally, the disclosed techniques may be integrated withoptimization (i.e., error reduction or minimization) algorithms thatallow hard constraints. In such implementations, the penalty frameworkmay not be discarded. Rather, hard constraints and penalties may be usedtogether in tandem to ensure desired interactions.

1-3. (canceled)
 4. A method of manipulating a three-dimensional objectdisplayed in a multi-touch display device, the method comprising:displaying the object in two dimensions on the display device, theobject having an initial three-dimensional location and an initialthree-dimensional rotational orientation; detecting one or more inputmechanisms touching one or more touch points on the display device;determining a two-dimensional location of each of the one or more touchpoints on the display device; determining, for each of the one or moretouch points, a three-dimensional contact point on a surface of theobject that corresponds to the two-dimensional location of the touchpoint; detecting a change in applied pressure by the one or more inputmechanisms at a selected one of the touch points that corresponds to aselected one of the contact points; determining a target depth value forthe selected contact point based on the change in applied pressure atthe selected touch point; calculating a three-dimensional transformationof the object by specifying at least one of a new three-dimensionalrotation and a new three-dimensional location for the object, andreducing a difference between a depth value of the selected contactpoint after the object transformation and the target depth value;transforming the object using the three-dimensional transformation suchthat the transformed object is positioned and rotated in accordance withthe at least one of the new three-dimensional location and the newthree-dimensional rotation; and displaying the transformedthree-dimensional object on the display device.
 5. The method of claim4, wherein displaying the object in two dimensions further comprisingprojecting the object onto an image plane of a camera, and the methodfurther comprises: accessing a current depth value of the selectedcontact point, the selected contact point being projected for displayonto the image plane of the camera at the location of the selected touchpoint, and the current depth value of the selected contact pointrepresenting a depth of the selected contact point along a Z-axis of thecamera away from the image plane.
 6. The method of claim 5, wherein theselected contact point is projected for display onto the image plane ofthe camera at the two-dimensional location of the selected touch point,and the transformed three-dimensional object is displayed by projectingthe transformed three-dimensional object onto the image plane of thecamera.
 7. The method of claim 4, further comprising reducing deviationbetween projected two-dimensional locations of the three-dimensionalcontact points after the object has been transformed and two dimensionallocations of the touch points corresponding to the contact points. 8.The method of claim 4, further comprising determining a surface area ofa contact area of the one or more touch points.
 9. The method of claim8, wherein detecting a change in the applied pressure is based on thesurface area of the contact area.
 10. The method of claim 8, wherein apressure sensor determines the surface area of the contact area of theone or more touch points, and the pressure sensor utilizes sensingtechnology selected from the group consisting of optical,electromagnetic, piezoelectric, and capacitive sensing technologies. 11.The method of claim 4, further comprising: for each of the one or moretouch points that is both stationary and at which a current pressure isgreater than a predetermined threshold, assigning a first predeterminedweight; for each of the one or more touch points that is not bothstationary and at which a current pressure is greater than apredetermined threshold, assigning a second predetermined weight that issmaller than the first predetermined weight; and controlling an errordistribution among the one or more touch points based at least in parton the first predetermined weight and the second predetermined weight.12. The method of claim 11, wherein controlling the error distributionfurther comprises allocating less error to each of the one or more touchpoints that is both stationary and at which the current pressure isgreater than the predetermined threshold as compared to each of the oneor more touch points that is not both stationary and at which a currentpressure is greater than a predetermined threshold.
 13. The method ofclaim 11, further comprising concurrently using one or more otherweighting schemes to assign the first predetermined weight and thesecond predetermined weight.
 14. A multi-touch display device,comprising: a processor; and a program of instructions executable by theprocessor to manipulate a three-dimensional object displayed by thedevice, the program of instructions configured to: display the object intwo dimensions on the display device, the object having an initialthree-dimensional location and an initial three-dimensional rotationalorientation; detect one or more input mechanisms touching one or moretouch points on the display device; determine a two-dimensional locationof each of the one or more touch points on the display device;determine, for each of the one or more touch points, a three-dimensionalcontact point on a surface of the object that corresponds to thetwo-dimensional location of the touch point; detect a change in appliedpressure by the one or more input mechanisms at a selected one of thetouch points that corresponds to a selected one of the contact points;determine a target depth value for the selected contact point based onthe change in applied pressure at the selected touch point; calculate athree-dimensional transformation of the object by specifying at leastone of a new three-dimensional rotation and a new three-dimensionallocation for the object, and reducing a difference between a depth valueof the selected contact point after the object transformation and thetarget depth value; transform the object using the three-dimensionaltransformation such that the transformed object is positioned androtated in accordance with the at least one of the new three-dimensionallocation and the new three-dimensional rotation; and display thetransformed three-dimensional object on the display device.
 15. Themulti-touch display device of claim 14, wherein the program ofinstructions is further configured to: display the object in twodimensions by projecting the object onto an image plane of a camera, andthe method further comprises; and access a current depth value of theselected contact point, the selected contact point being projected fordisplay onto the image plane of the camera at the location of theselected touch point, and the current depth value of the selectedcontact point representing a depth of the selected contact point along aZ-axis of the camera away from the image plane.
 16. The multi-touchdisplay device of claim 15, wherein the selected contact point isprojected for display onto the image plane of the camera at thetwo-dimensional location of the selected touch point, and thetransformed three-dimensional object is displayed by projecting thetransformed three-dimensional object onto the image plane of the camera.17. The multi-touch display device of claim 14, wherein the program ofinstructions is further configured to reduce deviation between projectedtwo-dimensional locations of the three-dimensional contact points afterthe object has been transformed and two dimensional locations of thetouch points corresponding to the contact points.
 18. The multi-touchdisplay device of claim 14, wherein the program of instructions isfurther configured to determine a surface area of a contact area of theone or more touch points.
 19. The multi-touch display device of claim18, wherein the program of instructions is further configured to detecta change in the applied pressure based on the surface area of thecontact area.
 20. The multi-touch display device of claim 18, furthercomprising a pressure sensor that determines the surface area of thecontact area of the one or more touch points, wherein the pressuresensor utilizes sensing technology selected from the group consisting ofoptical, electromagnetic, piezoelectric, and capacitive sensingtechnologies.
 21. The multi-touch display device of claim 14, whereinthe program of instructions is further configured to: for each of theone or more touch points that is both stationary and at which a currentpressure is greater than a predetermined threshold, assign a firstpredetermined weight; for each of the one or more touch points that isnot both stationary and at which a current pressure is greater than apredetermined threshold, assign a second predetermined weight that issmaller than the first predetermined weight; and control an errordistribution among the one or more touch points based at least in parton the first predetermined weight and the second predetermined weight.22. The multi-touch display device of claim 21, wherein the program ofinstructions is further configured to control the error distribution byallocating less error to each of the one or more touch points that isboth stationary and at which the current pressure is greater than thepredetermined threshold as compared to each of the one or more touchpoints that is not both stationary and at which a current pressure isgreater than a predetermined threshold.
 23. A method of manipulating athree-dimensional object displayed in a multi-touch display device, themethod comprising: displaying the object in two dimensions on thedisplay device, the object having an initial three-dimensional locationand an initial three-dimensional rotational orientation; detecting oneor more input mechanisms touching one or more touch points on thedisplay device; determining a two-dimensional location of each of theone or more touch points on the display device; determining, for each ofthe one or more touch points, a three-dimensional contact point on asurface of the object that corresponds to the two-dimensional locationof the touch point; detecting a change in applied pressure by the one ormore input mechanisms at a selected one of the touch points thatcorresponds to a selected one of the contact points; determining atarget depth value for the selected contact point based on the change inapplied pressure at the selected touch point; calculating athree-dimensional transformation of the object by specifying at leastone of a new three-dimensional rotation and a new three-dimensionallocation for the object, and by reducing a difference between a depthvalue of the selected contact point after the object transformation andthe target depth value; transforming the object using thethree-dimensional transformation such that the transformed object ispositioned and rotated in accordance with the at least one of the newthree-dimensional location and the new three-dimensional rotation;reducing deviation between projected two-dimensional locations of thethree-dimensional contact points after the object has been transformedand two dimensional locations of the touch points corresponding to thecontact points; and after the deviation has been reduced, displaying thetransformed three-dimensional object on the display device.